Sep 08 07:02:35.598944 ip-10-0-139-109 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 07:02:35.598957 ip-10-0-139-109 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 07:02:35.598968 ip-10-0-139-109 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 07:02:35.599201 ip-10-0-139-109 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 07:02:45.848162 ip-10-0-139-109 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 07:02:45.848181 ip-10-0-139-109 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ef47ca1751cd44ff9eaf2e01c974c094 --
Sep 08 07:04:43.049531 ip-10-0-139-109 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 07:04:43.446345 ip-10-0-139-109 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 07:04:43.446345 ip-10-0-139-109 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 07:04:43.446345 ip-10-0-139-109 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 07:04:43.446345 ip-10-0-139-109 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 07:04:43.446345 ip-10-0-139-109 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 07:04:43.447123 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.446386    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 07:04:43.452195 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452170    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 07:04:43.452195 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452190    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 07:04:43.452195 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452196    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 07:04:43.452195 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452200    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 07:04:43.452195 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452204    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452208    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452213    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452217    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452236    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452242    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452246    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452250    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452255    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452259    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452263    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452266    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452270    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452275    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452279    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452283    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452287    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452291    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452295    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452299    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 07:04:43.452582 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452303    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452307    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452312    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452316    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452330    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452334    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452338    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452342    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452346    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452349    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452353    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452357    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452361    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452365    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452371    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452376    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452380    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452384    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452389    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452393    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 07:04:43.453428 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452397    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452403    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452407    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452411    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452415    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452419    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452423    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452427    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452433    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452437    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452442    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452446    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452449    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452453    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452458    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452462    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452466    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452471    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452475    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 07:04:43.454307 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452482    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452489    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452494    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452500    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452504    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452508    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452515    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452521    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452525    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452530    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452534    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452538    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452543    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452547    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452569    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452576    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452581    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452585    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452589    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 07:04:43.455097 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452593    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452597    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452600    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452605    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452609    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452613    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452617    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452621    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452626    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452630    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452634    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452638    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452642    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452646    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452650    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452657    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.452663    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453921    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453932    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453938    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 07:04:43.455822 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453943    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453947    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453952    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453956    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453960    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453965    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453969    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453973    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453987    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453992    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.453996    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454000    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454005    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454009    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454013    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454017    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454021    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454025    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454029    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454033    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 07:04:43.456354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454037    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454041    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454045    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454049    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454054    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454058    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454062    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454065    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454070    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454074    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454080    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454086    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454091    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454095    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454099    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454103    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454107    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454111    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454115    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454119    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 07:04:43.456867 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454123    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454135    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454139    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454143    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454147    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454151    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454155    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454159    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454163    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454167    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454171    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454175    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454179    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454183    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454187    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454191    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454195    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454200    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454204    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454208    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 07:04:43.457389 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454212    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454218    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454240    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454247    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454253    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454257    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454261    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454265    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454269    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454272    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454276    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454281    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454285    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454289    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454303    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454309    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454314    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454318    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454323    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 07:04:43.457896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454327    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454331    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454334    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454339    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454342    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454347    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454351    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454356    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454360    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454364    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454368    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454372    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454376    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454382    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454386    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454391    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.454400    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455110    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455118    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 07:04:43.458390 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455124    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455128    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455135    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455141    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455146    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455151    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455155    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455159    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455165    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455169    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455174    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455178    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455183    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455188    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455192    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455196    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455200    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455204    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455208    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 07:04:43.458882 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455212    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455216    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455241    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455246    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455250    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455254    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455259    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455263    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455267    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455270    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455282    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455286    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455290    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455294    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455298    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455302    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455309    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455313    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455317    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455321    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 07:04:43.459396 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455325    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455330    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455335    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455339    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455344    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455348    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455352    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455356    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455361    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455365    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455369    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455373    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455377    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455381    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455385    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455389    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455393    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455397    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455401    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455405    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 07:04:43.459896 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455410    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455414    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455420    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455426    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455432    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455437    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455441    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455445    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455449    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455454    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455458    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455463    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455467    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455471    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455475    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455483    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455488    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455492    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455496    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455500    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 07:04:43.460416 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455504    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455510    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455515    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455521    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455526    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455530    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455535    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455539    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455544    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455548    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455552    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455556    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455560    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455565    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455571    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455575    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455579    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.455583    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455697    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455742    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455751    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 07:04:43.460923 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455756    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455761    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455764    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455769    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455773    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455777    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455780    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455783    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455787    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455790    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455793    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455796    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455799    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455801    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455804    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455809    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455811    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455815    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455818    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455821    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455825    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455828    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455832    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 07:04:43.461562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455835    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455839    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455842    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455844    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455848    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455851    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455855    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455861    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455864    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455867    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455870    2580 flags.go:64] FLAG: --enable-server="true"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455873    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455878    2580 flags.go:64] FLAG: --event-burst="100"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455881    2580 flags.go:64] FLAG: --event-qps="50"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455884    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455887    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455890    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455894    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455897    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455900    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455903    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455906    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455909    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455912    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455915    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 07:04:43.462125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455918    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455921    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455924    2580 flags.go:64] FLAG: --feature-gates=""
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455928    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455931    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455934    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455937    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455940    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455943    2580 flags.go:64] FLAG: --help="false"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455946    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455949    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455952    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455955    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455958    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455964    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455970    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455973    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455976    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455979    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455982    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455985    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455988    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455991    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455994    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 07:04:43.462830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455996    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.455999    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456003    2580 flags.go:64] FLAG: --lock-file=""
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456005    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456008    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456011    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456016    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456019    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456022    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456025    2580 flags.go:64] FLAG: --logging-format="text"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456028    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456031    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456034    2580 flags.go:64] FLAG: --manifest-url=""
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456037    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456041    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456044    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456049    2580 flags.go:64] FLAG: --max-pods="110"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456052    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456055    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456058    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456061    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456064    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456069    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456073    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456082    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 07:04:43.463478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456086    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456089    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456092    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456095    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456101    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456104    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456107    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456110    2580 flags.go:64] FLAG: --port="10250"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456113    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456116    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07ce93cb80940e0a3"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456119    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456122    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456126    2580 flags.go:64] FLAG: --register-node="true"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456128    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456132    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456135    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456138    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456141    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456145    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456148    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456152    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456154    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456157    2580 flags.go:64] FLAG: --runonce="false"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456160    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456163    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 07:04:43.464103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456166    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456169    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456172    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456175    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456178    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456184    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456187    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456190    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456193    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456196    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456199    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456202    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456205    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456211    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456215    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456218    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456236    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456239    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456242    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456245    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456248    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456251    2580 flags.go:64] FLAG: --v="2"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456255    2580 flags.go:64] FLAG: --version="false"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456259    2580 flags.go:64] FLAG: --vmodule=""
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456263    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 07:04:43.464730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.456266    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456389    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456393    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456397    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456400    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456402    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456406    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456408    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456411    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456414    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456419    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456421    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456426    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456429    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456432    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456435    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456438    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456441    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456444    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456447    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 07:04:43.465370 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456449    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456452    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456456    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456460    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456462    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456465    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456468    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456471    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456474    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456479    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456482    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456487    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456491    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456494    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456496    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456499    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456503    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456506    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456509    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 07:04:43.465873 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456512    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456515    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456518    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456522    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456525    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456528    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456531    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456534    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456537    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456541    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456543    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456546    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456548    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456552    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456555    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456558    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456561    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456564    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456566    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 07:04:43.466431 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456569    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456571    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456574    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456576    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456579    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456582    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456584    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456586    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456589    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456592    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456594    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456597    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456599    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456602    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456604    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456607    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456611    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456613    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456617    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456620    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 07:04:43.466904 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456622    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456625    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456628    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456631    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456634    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456637    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456639    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456642    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456645    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456647    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456650    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456652    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456655    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456657    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456660    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456662    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456665    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456667    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456670    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 07:04:43.467454 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456673    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456675    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.456678    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.457511    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.464553    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.464572    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.465330    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.465337    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465416    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465422    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465426    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465429    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465431    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465434    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465437    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 07:04:43.467937 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465440    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465448    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465451    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465453    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465458    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465462    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465465    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465468    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465471    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465474    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465477    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465481    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465490    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465493    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465496    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465499    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465502    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465505    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 07:04:43.468349 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465508    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465511    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465514    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465516    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465520    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465523    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465526    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465528    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465531    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465534    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465536    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465539    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465541    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465544    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465546    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465550    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465561    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465564    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465567    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 07:04:43.468800 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465569    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465572    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465575    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465577    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465580    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465583    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465585    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465588    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465591    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465593    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465596    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465599    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465601    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465604    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465606    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465609    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465612    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465615    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465618    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465621    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 07:04:43.469297 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465624    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465627    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465629    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465632    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465635    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465637    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465640    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465642    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465645    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465647    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465655    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465658    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465661    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465663    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465666    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465668    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465671    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465674    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465676    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465679    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 07:04:43.469785 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465682    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465684    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465687    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465690    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465692    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465695    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465697    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465700    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465702    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465706    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465709    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465712    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465714    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465717    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465720    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 07:04:43.470386 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.465724    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465873    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465877    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465880    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465883    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465886    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465889    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465892    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465900    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465903    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465905    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465908    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465911    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465913    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465916    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465918    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465921    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465923    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465926    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465928    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 07:04:43.470809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465931    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465933    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465936    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465939    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465941    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465943    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465946    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465949    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465952    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465955    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465957    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465960    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465962    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465965    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465968    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465970    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465973    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465975    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465978    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465980    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 07:04:43.471354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465984    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465992    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465995    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.465998    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466000    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466003    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466005    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466008    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466011    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466013    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466016    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466018    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466021    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466023    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466026    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466029    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466031    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466034    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466036    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466040    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 07:04:43.471854 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466042    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466045    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466048    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466050    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466053    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466055    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466057    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466060    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466063    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466065    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466069    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466073    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466076    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466079    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466087    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466090    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466092    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466095    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466098    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 07:04:43.472378 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466100    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466103    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466105    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466108    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466110    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466113    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466115    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466118    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466120    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466123    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466125    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466128    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466131    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466133    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466136    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466139    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466141    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466144    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466146    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 07:04:43.472859 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466149    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 07:04:43.473345 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:43.466153    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 07:04:43.473345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.466158    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 07:04:43.473345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.466324    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 07:04:43.473345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.470185    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 07:04:43.473345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.471110    2580 server.go:1019] "Starting client certificate rotation"
Sep 08 07:04:43.473345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.471208    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 07:04:43.473345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.471302    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 07:04:43.475612 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.475587    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 07:04:43.477914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.477892    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 07:04:43.490827 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.490812    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 08 07:04:43.493579 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.493563    2580 log.go:25] "Validated CRI v1 image API"
Sep 08 07:04:43.494356 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.494339    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 07:04:43.496740 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.496722    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 07:04:43.497495 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.497472    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 94daad29-cbd1-4d99-a239-cb3db3e9fae0:/dev/nvme0n1p4 d3457f6e-bc6d-4eb3-9e27-d2f48500fa0e:/dev/nvme0n1p3]
Sep 08 07:04:43.497562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.497495    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 07:04:43.502722 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.502614    2580 manager.go:216] Machine: {Timestamp:2026-09-08 07:04:43.501070784 +0000 UTC m=+0.335766870 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100769 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27a6022edb03302a6efb1ab345a47d SystemUUID:ec27a602-2edb-0330-2a6e-fb1ab345a47d BootID:ef47ca17-51cd-44ff-9eaf-2e01c974c094 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:5a:99:9e:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:5a:99:9e:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:c5:0f:27:5a:05 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 07:04:43.502722 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.502717    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 07:04:43.502838 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.502827    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 07:04:43.505726 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.505705    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 07:04:43.505917 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.505730    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-109.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 07:04:43.506710 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.506701    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 07:04:43.506748 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.506713    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 07:04:43.506748 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.506721    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 07:04:43.506748 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.506735    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 07:04:43.507876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.507866    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 07:04:43.511685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.511674    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 07:04:43.511803 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.511794    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 07:04:43.511835 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.511823    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 07:04:43.511835 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.511832    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 07:04:43.511888 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.511845    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 07:04:43.511888 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.511855    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 08 07:04:43.511888 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.511870    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 07:04:43.512346 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.512332    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 07:04:43.512422 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.512351    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 07:04:43.514817 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.514802    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 07:04:43.519632 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.519615    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 07:04:43.519709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.519647    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 07:04:43.519817 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.519807    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 07:04:43.519848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.519822    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 07:04:43.519848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.519829    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 07:04:43.519848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.519834    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 07:04:43.519848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.519842    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 07:04:43.519848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.519848    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 07:04:43.520478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.520468    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 07:04:43.520478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.520478    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 07:04:43.520542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.520485    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 07:04:43.520542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.520491    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 07:04:43.520542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.520500    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 07:04:43.520542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.520507    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 07:04:43.521082 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.521072    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 07:04:43.521082 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.521081    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 07:04:43.522674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.522656    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lt5r4"
Sep 08 07:04:43.522914 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.522893    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 07:04:43.523014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.522999    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-109.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 07:04:43.523255 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.523239    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 07:04:43.525093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.525079    2580 server.go:1268] "Started kubelet"
Sep 08 07:04:43.525192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.525159    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 07:04:43.525249 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.525183    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 07:04:43.525287 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.525263    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 07:04:43.526038 ip-10-0-139-109 systemd[1]: Started Kubernetes Kubelet.
Sep 08 07:04:43.526390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.526374    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 07:04:43.529453 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.529433    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lt5r4"
Sep 08 07:04:43.530102 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.530093    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 07:04:43.530739 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.530700    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 07:04:43.532458 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.532441    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 07:04:43.532535 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.532460    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 07:04:43.532585 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.532547    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:43.533050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533035    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 07:04:43.533167 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533036    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 07:04:43.533268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533257    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 07:04:43.533410 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533393    2580 factory.go:55] Registering systemd factory
Sep 08 07:04:43.533472 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533414    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 08 07:04:43.533547 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533518    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 07:04:43.533547 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533534    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 07:04:43.533692 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533663    2580 factory.go:153] Registering CRI-O factory
Sep 08 07:04:43.533692 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533677    2580 factory.go:223] Registration of the crio container factory successfully
Sep 08 07:04:43.533802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533719    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 07:04:43.533802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533737    2580 factory.go:103] Registering Raw factory
Sep 08 07:04:43.533802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.533751    2580 manager.go:1196] Started watching for new ooms in manager
Sep 08 07:04:43.534110 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.534098    2580 manager.go:349] Starting recovery of all containers
Sep 08 07:04:43.540883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.540837    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 07:04:43.543852 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.543831    2580 manager.go:354] Recovery completed
Sep 08 07:04:43.545238 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.545197    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-109.ec2.internal\" not found" node="ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.551598 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.551585    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 07:04:43.554519 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.554502    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 07:04:43.554589 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.554534    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 07:04:43.554589 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.554549    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientPID"
Sep 08 07:04:43.555162 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.555136    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 07:04:43.555162 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.555146    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 07:04:43.555162 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.555150    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 07:04:43.555319 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.555170    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 07:04:43.555883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.555868    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 07:04:43.555954 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.555893    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 07:04:43.555954 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.555904    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientPID"
Sep 08 07:04:43.556462 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.556446    2580 policy_none.go:49] "None policy: Start"
Sep 08 07:04:43.556462 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.556464    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 07:04:43.556551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.556474    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 07:04:43.557482 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.557471    2580 policy_none.go:47] "Start"
Sep 08 07:04:43.608854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.608832    2580 manager.go:337] "Starting Device Plugin manager"
Sep 08 07:04:43.608973 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.608886    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 07:04:43.608973 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.608899    2580 server.go:87] "Starting device plugin registration server"
Sep 08 07:04:43.609198 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.609187    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 07:04:43.609256 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.609201    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 07:04:43.609351 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.609332    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 07:04:43.609476 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.609414    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 07:04:43.609476 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.609424    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 07:04:43.609982 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.609957    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 07:04:43.610069 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.610010    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:43.630757 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.630615    2580 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2613 into the system container \"/system.slice\": <nil>"
Sep 08 07:04:43.709448 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.709426    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 07:04:43.720548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.710427    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 07:04:43.720548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.710460    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 07:04:43.720548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.710475    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientPID"
Sep 08 07:04:43.720548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.710502    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.720548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.718996    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.720548 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.719016    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-109.ec2.internal\": node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:43.731583 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.731562    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:43.781236 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.781174    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 07:04:43.782444 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.782425    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 07:04:43.782495 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.782479    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 07:04:43.782532 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.782503    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 07:04:43.782643 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.782603    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 07:04:43.784856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.784839    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 07:04:43.831718 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.831688    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:43.883120 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.883077    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal"]
Sep 08 07:04:43.883290 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.883168    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 07:04:43.884346 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.884329    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 07:04:43.884426 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.884358    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 07:04:43.884426 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.884368    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientPID"
Sep 08 07:04:43.885891 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.885878    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 07:04:43.885998 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.885983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.886038 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.886029    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 07:04:43.886738 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.886721    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 07:04:43.886831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.886744    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 07:04:43.886831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.886758    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientPID"
Sep 08 07:04:43.886831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.886778    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 07:04:43.886831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.886798    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 07:04:43.886831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.886807    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientPID"
Sep 08 07:04:43.887988 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.887972    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.888082 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.887997    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 07:04:43.889004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.888987    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 07:04:43.889091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.889019    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 07:04:43.889091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.889034    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeHasSufficientPID"
Sep 08 07:04:43.911966 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.911945    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-109.ec2.internal\" not found" node="ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.916326 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.916310    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-109.ec2.internal\" not found" node="ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.931990 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:43.931974    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:43.935286 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.935263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/901888addc1cc153e7772482f8ca35e4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-109.ec2.internal\" (UID: \"901888addc1cc153e7772482f8ca35e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.935358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.935293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c2563eab4d47e90fa012e780aa92e92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal\" (UID: \"4c2563eab4d47e90fa012e780aa92e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:43.935358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:43.935312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c2563eab4d47e90fa012e780aa92e92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal\" (UID: \"4c2563eab4d47e90fa012e780aa92e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:44.033028 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:44.032925    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:44.036204 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:44.036186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c2563eab4d47e90fa012e780aa92e92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal\" (UID: \"4c2563eab4d47e90fa012e780aa92e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:44.036284 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:44.036212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c2563eab4d47e90fa012e780aa92e92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal\" (UID: \"4c2563eab4d47e90fa012e780aa92e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:44.036284 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:44.036246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/901888addc1cc153e7772482f8ca35e4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-109.ec2.internal\" (UID: \"901888addc1cc153e7772482f8ca35e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:44.036353 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:44.036289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c2563eab4d47e90fa012e780aa92e92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal\" (UID: \"4c2563eab4d47e90fa012e780aa92e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:44.036353 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:44.036341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c2563eab4d47e90fa012e780aa92e92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal\" (UID: \"4c2563eab4d47e90fa012e780aa92e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:44.036411 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:44.036369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/901888addc1cc153e7772482f8ca35e4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-109.ec2.internal\" (UID: \"901888addc1cc153e7772482f8ca35e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:45.284642 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:45.284618    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:45.365910 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.365866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:45.370677 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.370654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:45.385415 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:45.385393    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:45.485958 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:45.485873    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:45.586387 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:45.586360    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:45.622715 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.622677    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 07:04:45.623413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.622858    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 07:04:45.623413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.622885    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 07:04:45.684680 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.684637    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 06:59:44 +0000 UTC" deadline="2028-02-15 18:55:33.640729915 +0000 UTC"
Sep 08 07:04:45.684680 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.684670    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12611h50m47.956064423s"
Sep 08 07:04:45.686741 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:45.686724    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:45.708253 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.708226    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 07:04:45.787189 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:45.787095    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:45.887595 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:45.887564    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-109.ec2.internal\" not found"
Sep 08 07:04:45.920079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.920053    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 07:04:45.984508 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.984478    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:45.997560 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.997541    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 07:04:45.999128 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:45.999107    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal"
Sep 08 07:04:46.006824 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.006809    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 07:04:46.127201 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:46.127166    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901888addc1cc153e7772482f8ca35e4.slice/crio-eb190e8cb76b4039dbd93ee361c6368245ce8f0a02d2c9fa712398b46224e0a8 WatchSource:0}: Error finding container eb190e8cb76b4039dbd93ee361c6368245ce8f0a02d2c9fa712398b46224e0a8: Status 404 returned error can't find the container with id eb190e8cb76b4039dbd93ee361c6368245ce8f0a02d2c9fa712398b46224e0a8
Sep 08 07:04:46.127573 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:46.127554    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2563eab4d47e90fa012e780aa92e92.slice/crio-9b59a922629676a8e21815181cdcee4955c20942ae7bd87c3442be462962604d WatchSource:0}: Error finding container 9b59a922629676a8e21815181cdcee4955c20942ae7bd87c3442be462962604d: Status 404 returned error can't find the container with id 9b59a922629676a8e21815181cdcee4955c20942ae7bd87c3442be462962604d
Sep 08 07:04:46.132204 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.132187    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 07:04:46.664091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.664043    2580 apiserver.go:52] "Watching apiserver"
Sep 08 07:04:46.673103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.673054    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 07:04:46.673584 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.673562    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5","openshift-cluster-node-tuning-operator/tuned-rlrr9","openshift-dns/node-resolver-v95nw","openshift-image-registry/node-ca-lc7d6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal","openshift-multus/multus-additional-cni-plugins-8nvwv","openshift-multus/multus-qwkps","kube-system/global-pull-secret-syncer-zrqsh","kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal","openshift-multus/network-metrics-daemon-hb6qs","openshift-network-diagnostics/network-check-target-gtz5m","openshift-network-operator/iptables-alerter-nhxll","openshift-ovn-kubernetes/ovnkube-node-zpbjh","kube-system/konnectivity-agent-6g8t9"]
Sep 08 07:04:46.675431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.675404    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:46.675549 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.675527    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:04:46.676686 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.676666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.678471 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.677925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:04:46.678471 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.678435    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wmv2r\""
Sep 08 07:04:46.678619 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.678583    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 07:04:46.678889 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.678867    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 07:04:46.679815 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.679393    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:46.679815 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.679447    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:04:46.679815 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.679632    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 07:04:46.680295 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.680245    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 07:04:46.680674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.680658    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n5zqc\""
Sep 08 07:04:46.680870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.680855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.681814 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.681794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.682690 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.682665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 07:04:46.683100 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.682968    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 07:04:46.683100 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.682986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 07:04:46.684400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.683537    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bv9m8\""
Sep 08 07:04:46.684400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.683756    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 07:04:46.684400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.683978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.684400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.684111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 07:04:46.684400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.684112    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 07:04:46.684400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.684320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n8wtf\""
Sep 08 07:04:46.684812 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.684725    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 07:04:46.685358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.685339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.686683 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.686511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 07:04:46.686683 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.686543    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 07:04:46.687250 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.686908    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 07:04:46.687250 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.686917    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 07:04:46.687250 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.686993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.688405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.688385    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-78gfd\""
Sep 08 07:04:46.688512 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.688429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 07:04:46.688577 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.688535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 07:04:46.689028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.688645    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 07:04:46.689028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.688857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 07:04:46.689028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.688890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 07:04:46.689028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.688907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kjbvg\""
Sep 08 07:04:46.689278 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.689108    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.689801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.689621    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 07:04:46.689801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.689655    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 07:04:46.689801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.689683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7g8gw\""
Sep 08 07:04:46.689801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.689786    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 07:04:46.691367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.690832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.691367 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.690892    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:04:46.691367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.691213    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 07:04:46.691367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.691278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n8jxb\""
Sep 08 07:04:46.691774 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.691654    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 07:04:46.691863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.691794    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 07:04:46.692639 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.692617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.694176 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpn2h\" (UniqueName: \"kubernetes.io/projected/18397989-832c-4e4d-b309-f09e6c5e0207-kube-api-access-xpn2h\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:46.694275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysctl-d\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.694275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysctl-conf\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.694378 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-cnibin\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.694378 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-system-cni-dir\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.694378 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-cni-bin\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.694378 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.694378 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694374    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 07:04:46.694558 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysconfig\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.694558 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/821a0f41-2105-4003-b56f-17eec112c8bd-etc-tuned\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.694558 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-system-cni-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.695811 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.694529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-k8s-cni-cncf-io\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.695898 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.695848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-etc-kubernetes\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.695898 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.695883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-ovn\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.696001 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.695941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-lib-modules\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.696001 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.695969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-systemd-units\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-systemd\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rwxn\""
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-conf-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bts45\" (UniqueName: \"kubernetes.io/projected/8b7a2c1b-1105-437c-bdca-c0e6495ff408-kube-api-access-bts45\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40e80d3a-bc71-4e0b-8616-f022edcc6fcb-konnectivity-ca\") pod \"konnectivity-agent-6g8t9\" (UID: \"40e80d3a-bc71-4e0b-8616-f022edcc6fcb\") " pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-hostroot\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-ovnkube-script-lib\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.696553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbqt7\" (UniqueName: \"kubernetes.io/projected/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-kube-api-access-qbqt7\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.697215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.697021    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 07:04:46.698856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.698835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-run\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.698944 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.698877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:46.698944 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.698910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-cni-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.699079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.698975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-daemon-config\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.699079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-run-ovn-kubernetes\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-cni-netd\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-sys\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.699271 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-socket-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.699271 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-cni-multus\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.699271 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-multus-certs\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.699271 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-var-lib-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699271 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-env-overrides\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40e80d3a-bc71-4e0b-8616-f022edcc6fcb-agent-certs\") pod \"konnectivity-agent-6g8t9\" (UID: \"40e80d3a-bc71-4e0b-8616-f022edcc6fcb\") " pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:04:46.699511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-systemd\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-node-log\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.699511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-kubelet\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.699747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-kubelet\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c075acc-7e18-42fb-8778-e05d68b02569-ovn-node-metrics-cert\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzqg\" (UniqueName: \"kubernetes.io/projected/0c075acc-7e18-42fb-8778-e05d68b02569-kube-api-access-vbzqg\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.699747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-var-lib-kubelet\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.699747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-host\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.699967 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/821a0f41-2105-4003-b56f-17eec112c8bd-tmp\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.699967 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cnibin\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.699967 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-kubernetes\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.699967 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b7a2c1b-1105-437c-bdca-c0e6495ff408-cni-binary-copy\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.699967 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkpl\" (UniqueName: \"kubernetes.io/projected/821a0f41-2105-4003-b56f-17eec112c8bd-kube-api-access-mwkpl\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.700210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.699941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-sys-fs\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.700210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-ovnkube-config\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.700210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.700210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.700210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-slash\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.700438 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786f8\" (UniqueName: \"kubernetes.io/projected/e1280dda-1e18-457e-b25c-b9bd2f5cd880-kube-api-access-786f8\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.700438 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:46.700438 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700302    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 07:04:46.700438 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-modprobe-d\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.700623 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.700679 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-etc-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.700727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-log-socket\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.700778 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-cni-bin\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.700778 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.700903 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-run-netns\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.700903 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-registration-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.700903 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-device-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.701062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-socket-dir-parent\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.701062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.700956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-netns\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.728334 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.728312    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rcmwx"
Sep 08 07:04:46.736333 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.736308    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rcmwx"
Sep 08 07:04:46.785721 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.785691    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 07:04:46.801585 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-socket-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.801737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-cni-multus\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.801737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-multus-certs\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.801737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-var-lib-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.801737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-env-overrides\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.801737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-cni-multus\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.801737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-multus-certs\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.802045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-kubelet-config\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.802045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-var-lib-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.802045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-socket-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.802045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.801971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40e80d3a-bc71-4e0b-8616-f022edcc6fcb-agent-certs\") pod \"konnectivity-agent-6g8t9\" (UID: \"40e80d3a-bc71-4e0b-8616-f022edcc6fcb\") " pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:04:46.802045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-systemd\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-node-log\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-systemd\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-kubelet\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-kubelet\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c075acc-7e18-42fb-8778-e05d68b02569-ovn-node-metrics-cert\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-node-log\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzqg\" (UniqueName: \"kubernetes.io/projected/0c075acc-7e18-42fb-8778-e05d68b02569-kube-api-access-vbzqg\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d745d326-a791-4108-b6bd-59caff1b804b-hosts-file\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e4a766e-393c-42b7-912b-5aabacc745f4-serviceca\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hclf\" (UniqueName: \"kubernetes.io/projected/3e4a766e-393c-42b7-912b-5aabacc745f4-kube-api-access-8hclf\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-var-lib-kubelet\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-host\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-kubelet\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-kubelet\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-host\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-var-lib-kubelet\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802439    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 07:04:46.802477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/821a0f41-2105-4003-b56f-17eec112c8bd-tmp\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cnibin\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-host-slash\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-kubernetes\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b7a2c1b-1105-437c-bdca-c0e6495ff408-cni-binary-copy\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkpl\" (UniqueName: \"kubernetes.io/projected/821a0f41-2105-4003-b56f-17eec112c8bd-kube-api-access-mwkpl\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-sys-fs\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cnibin\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-ovnkube-config\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d745d326-a791-4108-b6bd-59caff1b804b-tmp-dir\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-slash\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786f8\" (UniqueName: \"kubernetes.io/projected/e1280dda-1e18-457e-b25c-b9bd2f5cd880-kube-api-access-786f8\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-modprobe-d\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.803390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-env-overrides\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-kubernetes\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-sys-fs\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.802447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-slash\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-etc-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-log-socket\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-ovnkube-config\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-log-socket\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-cni-bin\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-var-lib-cni-bin\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b7a2c1b-1105-437c-bdca-c0e6495ff408-cni-binary-copy\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-etc-openvswitch\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-run-netns\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.804274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-registration-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-modprobe-d\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-device-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-socket-dir-parent\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-netns\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.803726    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4a766e-393c-42b7-912b-5aabacc745f4-host\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-run-netns\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpn2h\" (UniqueName: \"kubernetes.io/projected/18397989-832c-4e4d-b309-f09e6c5e0207-kube-api-access-xpn2h\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.803818    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs podName:18397989-832c-4e4d-b309-f09e6c5e0207 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:47.303776182 +0000 UTC m=+2.987236332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs") pod "network-metrics-daemon-hb6qs" (UID: "18397989-832c-4e4d-b309-f09e6c5e0207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-device-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-socket-dir-parent\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-netns\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysctl-d\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysctl-conf\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.803982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.805079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-cnibin\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysctl-d\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-system-cni-dir\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-registration-dir\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-system-cni-dir\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-cni-bin\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysctl-conf\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-cnibin\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-cni-bin\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysconfig\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/821a0f41-2105-4003-b56f-17eec112c8bd-etc-tuned\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-system-cni-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-k8s-cni-cncf-io\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-etc-kubernetes\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-sysconfig\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.805956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-ovn\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-system-cni-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-run-ovn\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-dbus\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-host-run-k8s-cni-cncf-io\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-lib-modules\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-etc-kubernetes\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-systemd-units\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-iptables-alerter-script\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26xmz\" (UniqueName: \"kubernetes.io/projected/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-kube-api-access-26xmz\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-lib-modules\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-systemd-units\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-systemd\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-conf-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-etc-systemd\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bts45\" (UniqueName: \"kubernetes.io/projected/8b7a2c1b-1105-437c-bdca-c0e6495ff408-kube-api-access-bts45\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-conf-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40e80d3a-bc71-4e0b-8616-f022edcc6fcb-konnectivity-ca\") pod \"konnectivity-agent-6g8t9\" (UID: \"40e80d3a-bc71-4e0b-8616-f022edcc6fcb\") " pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:04:46.806787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1280dda-1e18-457e-b25c-b9bd2f5cd880-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-hostroot\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-hostroot\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-ovnkube-script-lib\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zsk\" (UniqueName: \"kubernetes.io/projected/d745d326-a791-4108-b6bd-59caff1b804b-kube-api-access-k4zsk\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.804975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbqt7\" (UniqueName: \"kubernetes.io/projected/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-kube-api-access-qbqt7\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-run\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-cni-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-daemon-config\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-run-ovn-kubernetes\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-cni-netd\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-sys\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-sys\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40e80d3a-bc71-4e0b-8616-f022edcc6fcb-konnectivity-ca\") pod \"konnectivity-agent-6g8t9\" (UID: \"40e80d3a-bc71-4e0b-8616-f022edcc6fcb\") " pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:04:46.807688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-cni-dir\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c075acc-7e18-42fb-8778-e05d68b02569-ovnkube-script-lib\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/821a0f41-2105-4003-b56f-17eec112c8bd-run\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-run-ovn-kubernetes\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.805787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c075acc-7e18-42fb-8778-e05d68b02569-host-cni-netd\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.806028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b7a2c1b-1105-437c-bdca-c0e6495ff408-multus-daemon-config\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.806683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40e80d3a-bc71-4e0b-8616-f022edcc6fcb-agent-certs\") pod \"konnectivity-agent-6g8t9\" (UID: \"40e80d3a-bc71-4e0b-8616-f022edcc6fcb\") " pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.806874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/821a0f41-2105-4003-b56f-17eec112c8bd-etc-tuned\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.806936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c075acc-7e18-42fb-8778-e05d68b02569-ovn-node-metrics-cert\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.808469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.807000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/821a0f41-2105-4003-b56f-17eec112c8bd-tmp\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.815401 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.815372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkpl\" (UniqueName: \"kubernetes.io/projected/821a0f41-2105-4003-b56f-17eec112c8bd-kube-api-access-mwkpl\") pod \"tuned-rlrr9\" (UID: \"821a0f41-2105-4003-b56f-17eec112c8bd\") " pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.815542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.815524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-786f8\" (UniqueName: \"kubernetes.io/projected/e1280dda-1e18-457e-b25c-b9bd2f5cd880-kube-api-access-786f8\") pod \"aws-ebs-csi-driver-node-gjcf5\" (UID: \"e1280dda-1e18-457e-b25c-b9bd2f5cd880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:46.815654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.815635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzqg\" (UniqueName: \"kubernetes.io/projected/0c075acc-7e18-42fb-8778-e05d68b02569-kube-api-access-vbzqg\") pod \"ovnkube-node-zpbjh\" (UID: \"0c075acc-7e18-42fb-8778-e05d68b02569\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:46.815866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.815829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpn2h\" (UniqueName: \"kubernetes.io/projected/18397989-832c-4e4d-b309-f09e6c5e0207-kube-api-access-xpn2h\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:46.816937 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.816868    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 07:04:46.816937 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.816887    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 07:04:46.816937 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.816897    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hmtjs for pod openshift-network-diagnostics/network-check-target-gtz5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:46.817128 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.816953    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs podName:34662fee-4240-4d04-8694-a55dc1510dc4 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:47.316935633 +0000 UTC m=+3.000395766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hmtjs" (UniqueName: "kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs") pod "network-check-target-gtz5m" (UID: "34662fee-4240-4d04-8694-a55dc1510dc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:46.818145 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.818124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bts45\" (UniqueName: \"kubernetes.io/projected/8b7a2c1b-1105-437c-bdca-c0e6495ff408-kube-api-access-bts45\") pod \"multus-qwkps\" (UID: \"8b7a2c1b-1105-437c-bdca-c0e6495ff408\") " pod="openshift-multus/multus-qwkps"
Sep 08 07:04:46.818882 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.818861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbqt7\" (UniqueName: \"kubernetes.io/projected/37fef4ff-7a57-420d-ae7b-0dfd37943b3f-kube-api-access-qbqt7\") pod \"multus-additional-cni-plugins-8nvwv\" (UID: \"37fef4ff-7a57-420d-ae7b-0dfd37943b3f\") " pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:46.852705 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.852609    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 07:04:46.906408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-dbus\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.906408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-iptables-alerter-script\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.906654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xmz\" (UniqueName: \"kubernetes.io/projected/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-kube-api-access-26xmz\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.906654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zsk\" (UniqueName: \"kubernetes.io/projected/d745d326-a791-4108-b6bd-59caff1b804b-kube-api-access-k4zsk\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.906654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-kubelet-config\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.906654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.906654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d745d326-a791-4108-b6bd-59caff1b804b-hosts-file\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.906654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e4a766e-393c-42b7-912b-5aabacc745f4-serviceca\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.906654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hclf\" (UniqueName: \"kubernetes.io/projected/3e4a766e-393c-42b7-912b-5aabacc745f4-kube-api-access-8hclf\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.906654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-host-slash\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.907021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d745d326-a791-4108-b6bd-59caff1b804b-tmp-dir\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.907021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4a766e-393c-42b7-912b-5aabacc745f4-host\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.907021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4a766e-393c-42b7-912b-5aabacc745f4-host\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.907021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.906857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-dbus\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.907310 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.907280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d745d326-a791-4108-b6bd-59caff1b804b-hosts-file\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.907447 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.907351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-kubelet-config\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:46.907447 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.907379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-host-slash\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.907447 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.907388    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:46.907447 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.907414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-iptables-alerter-script\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.907669 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:46.907478    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret podName:bb1a056e-d8cc-4090-96ef-c2f53cfa99a8 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:47.407441108 +0000 UTC m=+3.090901240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret") pod "global-pull-secret-syncer-zrqsh" (UID: "bb1a056e-d8cc-4090-96ef-c2f53cfa99a8") : object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:46.907740 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.907718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d745d326-a791-4108-b6bd-59caff1b804b-tmp-dir\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.907875 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.907847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e4a766e-393c-42b7-912b-5aabacc745f4-serviceca\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.917818 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.917737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xmz\" (UniqueName: \"kubernetes.io/projected/f6b2b96b-9a18-4ba9-b565-d32aafb3cb07-kube-api-access-26xmz\") pod \"iptables-alerter-nhxll\" (UID: \"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07\") " pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:46.918239 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.918214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hclf\" (UniqueName: \"kubernetes.io/projected/3e4a766e-393c-42b7-912b-5aabacc745f4-kube-api-access-8hclf\") pod \"node-ca-lc7d6\" (UID: \"3e4a766e-393c-42b7-912b-5aabacc745f4\") " pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:46.918973 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.918944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zsk\" (UniqueName: \"kubernetes.io/projected/d745d326-a791-4108-b6bd-59caff1b804b-kube-api-access-k4zsk\") pod \"node-resolver-v95nw\" (UID: \"d745d326-a791-4108-b6bd-59caff1b804b\") " pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:46.940038 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.939985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal" event={"ID":"901888addc1cc153e7772482f8ca35e4","Type":"ContainerStarted","Data":"eb190e8cb76b4039dbd93ee361c6368245ce8f0a02d2c9fa712398b46224e0a8"}
Sep 08 07:04:46.941574 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.941549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal" event={"ID":"4c2563eab4d47e90fa012e780aa92e92","Type":"ContainerStarted","Data":"9b59a922629676a8e21815181cdcee4955c20942ae7bd87c3442be462962604d"}
Sep 08 07:04:46.991080 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.991049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rlrr9"
Sep 08 07:04:46.999638 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:46.999609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:04:47.002884 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:47.002852    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821a0f41_2105_4003_b56f_17eec112c8bd.slice/crio-91fd0fb8ec20447983c6eaa99d5e8abd790840152f8df9b84dd65c43947222f7 WatchSource:0}: Error finding container 91fd0fb8ec20447983c6eaa99d5e8abd790840152f8df9b84dd65c43947222f7: Status 404 returned error can't find the container with id 91fd0fb8ec20447983c6eaa99d5e8abd790840152f8df9b84dd65c43947222f7
Sep 08 07:04:47.010043 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.010020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qwkps"
Sep 08 07:04:47.012597 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:47.012563    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e80d3a_bc71_4e0b_8616_f022edcc6fcb.slice/crio-95e828ca91bcfbf6fad201e312816cb3b604dc53f56917d0291d1a1918f7792b WatchSource:0}: Error finding container 95e828ca91bcfbf6fad201e312816cb3b604dc53f56917d0291d1a1918f7792b: Status 404 returned error can't find the container with id 95e828ca91bcfbf6fad201e312816cb3b604dc53f56917d0291d1a1918f7792b
Sep 08 07:04:47.016096 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.016076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8nvwv"
Sep 08 07:04:47.019173 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:47.019146    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7a2c1b_1105_437c_bdca_c0e6495ff408.slice/crio-67de39c0b3aa0883898f4689e6571e4d6d5e723ef358172b237b796adef5f763 WatchSource:0}: Error finding container 67de39c0b3aa0883898f4689e6571e4d6d5e723ef358172b237b796adef5f763: Status 404 returned error can't find the container with id 67de39c0b3aa0883898f4689e6571e4d6d5e723ef358172b237b796adef5f763
Sep 08 07:04:47.023107 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.023083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:04:47.024971 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:47.024938    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fef4ff_7a57_420d_ae7b_0dfd37943b3f.slice/crio-3be99b73da060f9ddf6f3d0daba64920f952ffe3df4168e8543b22a422c5543c WatchSource:0}: Error finding container 3be99b73da060f9ddf6f3d0daba64920f952ffe3df4168e8543b22a422c5543c: Status 404 returned error can't find the container with id 3be99b73da060f9ddf6f3d0daba64920f952ffe3df4168e8543b22a422c5543c
Sep 08 07:04:47.031752 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.031732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5"
Sep 08 07:04:47.032863 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:47.032836    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c075acc_7e18_42fb_8778_e05d68b02569.slice/crio-f1357ec79cc26330710a6e45e62050a99b8d271542aa1bb286f3233094efc00b WatchSource:0}: Error finding container f1357ec79cc26330710a6e45e62050a99b8d271542aa1bb286f3233094efc00b: Status 404 returned error can't find the container with id f1357ec79cc26330710a6e45e62050a99b8d271542aa1bb286f3233094efc00b
Sep 08 07:04:47.039426 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.039266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lc7d6"
Sep 08 07:04:47.041604 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:47.041586    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1280dda_1e18_457e_b25c_b9bd2f5cd880.slice/crio-cec8e5011d951a4db1b0d53730d0fbbcdec06c1293525e2b227e9148d875e218 WatchSource:0}: Error finding container cec8e5011d951a4db1b0d53730d0fbbcdec06c1293525e2b227e9148d875e218: Status 404 returned error can't find the container with id cec8e5011d951a4db1b0d53730d0fbbcdec06c1293525e2b227e9148d875e218
Sep 08 07:04:47.048718 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.048560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nhxll"
Sep 08 07:04:47.053874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.053855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v95nw"
Sep 08 07:04:47.058376 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:47.058351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b2b96b_9a18_4ba9_b565_d32aafb3cb07.slice/crio-f07874d4b45a9d4a1b97a79112f6ffdd9f8d4dbd663437773dc028fecadfe597 WatchSource:0}: Error finding container f07874d4b45a9d4a1b97a79112f6ffdd9f8d4dbd663437773dc028fecadfe597: Status 404 returned error can't find the container with id f07874d4b45a9d4a1b97a79112f6ffdd9f8d4dbd663437773dc028fecadfe597
Sep 08 07:04:47.063374 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:04:47.063349    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd745d326_a791_4108_b6bd_59caff1b804b.slice/crio-8db9bf105df3e103e190fac909a071b819ff67529a48f55bf29d03c780587cca WatchSource:0}: Error finding container 8db9bf105df3e103e190fac909a071b819ff67529a48f55bf29d03c780587cca: Status 404 returned error can't find the container with id 8db9bf105df3e103e190fac909a071b819ff67529a48f55bf29d03c780587cca
Sep 08 07:04:47.178386 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.178283    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 07:04:47.309724 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.309687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:47.309897 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.309841    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:47.309950 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.309919    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs podName:18397989-832c-4e4d-b309-f09e6c5e0207 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:48.309898875 +0000 UTC m=+3.993359031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs") pod "network-metrics-daemon-hb6qs" (UID: "18397989-832c-4e4d-b309-f09e6c5e0207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:47.410700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.410663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:47.410855 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.410719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:47.410855 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.410833    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:47.410943 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.410859    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 07:04:47.410943 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.410880    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 07:04:47.410943 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.410892    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hmtjs for pod openshift-network-diagnostics/network-check-target-gtz5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:47.410943 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.410897    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret podName:bb1a056e-d8cc-4090-96ef-c2f53cfa99a8 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:48.410876609 +0000 UTC m=+4.094336746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret") pod "global-pull-secret-syncer-zrqsh" (UID: "bb1a056e-d8cc-4090-96ef-c2f53cfa99a8") : object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:47.411121 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.410945    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs podName:34662fee-4240-4d04-8694-a55dc1510dc4 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:48.410927536 +0000 UTC m=+4.094387674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmtjs" (UniqueName: "kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs") pod "network-check-target-gtz5m" (UID: "34662fee-4240-4d04-8694-a55dc1510dc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:47.739072 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.739031    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 06:59:46 +0000 UTC" deadline="2028-06-04 14:33:28.653963606 +0000 UTC"
Sep 08 07:04:47.739072 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.739069    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15247h28m40.914898952s"
Sep 08 07:04:47.934321 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.934283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:47.934514 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.934408    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:04:47.934813 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.934797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:47.934911 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:47.934891    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:04:47.946217 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.946144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nhxll" event={"ID":"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07","Type":"ContainerStarted","Data":"f07874d4b45a9d4a1b97a79112f6ffdd9f8d4dbd663437773dc028fecadfe597"}
Sep 08 07:04:47.952004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.951946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lc7d6" event={"ID":"3e4a766e-393c-42b7-912b-5aabacc745f4","Type":"ContainerStarted","Data":"0fa9d05691c5d16b209d53ae0496f09bb611dc52e06d7bd417a491f6c0b1fe16"}
Sep 08 07:04:47.953615 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.953536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5" event={"ID":"e1280dda-1e18-457e-b25c-b9bd2f5cd880","Type":"ContainerStarted","Data":"cec8e5011d951a4db1b0d53730d0fbbcdec06c1293525e2b227e9148d875e218"}
Sep 08 07:04:47.955785 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.955756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6g8t9" event={"ID":"40e80d3a-bc71-4e0b-8616-f022edcc6fcb","Type":"ContainerStarted","Data":"95e828ca91bcfbf6fad201e312816cb3b604dc53f56917d0291d1a1918f7792b"}
Sep 08 07:04:47.957513 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.957486    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rlrr9" event={"ID":"821a0f41-2105-4003-b56f-17eec112c8bd","Type":"ContainerStarted","Data":"91fd0fb8ec20447983c6eaa99d5e8abd790840152f8df9b84dd65c43947222f7"}
Sep 08 07:04:47.962242 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.962172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v95nw" event={"ID":"d745d326-a791-4108-b6bd-59caff1b804b","Type":"ContainerStarted","Data":"8db9bf105df3e103e190fac909a071b819ff67529a48f55bf29d03c780587cca"}
Sep 08 07:04:47.965174 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.965067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"f1357ec79cc26330710a6e45e62050a99b8d271542aa1bb286f3233094efc00b"}
Sep 08 07:04:47.970543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.970517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" event={"ID":"37fef4ff-7a57-420d-ae7b-0dfd37943b3f","Type":"ContainerStarted","Data":"3be99b73da060f9ddf6f3d0daba64920f952ffe3df4168e8543b22a422c5543c"}
Sep 08 07:04:47.973594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:47.972080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qwkps" event={"ID":"8b7a2c1b-1105-437c-bdca-c0e6495ff408","Type":"ContainerStarted","Data":"67de39c0b3aa0883898f4689e6571e4d6d5e723ef358172b237b796adef5f763"}
Sep 08 07:04:48.317813 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:48.317771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:48.317986 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.317916    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:48.317986 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.317978    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs podName:18397989-832c-4e4d-b309-f09e6c5e0207 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:50.317959486 +0000 UTC m=+6.001419620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs") pod "network-metrics-daemon-hb6qs" (UID: "18397989-832c-4e4d-b309-f09e6c5e0207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:48.418801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:48.418763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:48.418986 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:48.418819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:48.418986 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.418967    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:48.419104 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.419027    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret podName:bb1a056e-d8cc-4090-96ef-c2f53cfa99a8 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:50.419008718 +0000 UTC m=+6.102468857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret") pod "global-pull-secret-syncer-zrqsh" (UID: "bb1a056e-d8cc-4090-96ef-c2f53cfa99a8") : object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:48.419433 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.419415    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 07:04:48.419546 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.419443    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 07:04:48.419546 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.419471    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hmtjs for pod openshift-network-diagnostics/network-check-target-gtz5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:48.419546 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.419524    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs podName:34662fee-4240-4d04-8694-a55dc1510dc4 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:50.419497745 +0000 UTC m=+6.102957881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmtjs" (UniqueName: "kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs") pod "network-check-target-gtz5m" (UID: "34662fee-4240-4d04-8694-a55dc1510dc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:48.739454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:48.739382    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 06:59:46 +0000 UTC" deadline="2028-03-03 15:29:36.604317918 +0000 UTC"
Sep 08 07:04:48.739454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:48.739421    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13016h24m47.864901252s"
Sep 08 07:04:48.944845 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:48.943566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:48.944845 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:48.943736    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:04:49.934365 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:49.934282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:49.934784 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:49.934414    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:04:49.934829 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:49.934808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:49.934934 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:49.934910    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:04:50.334263 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:50.334224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:50.334449 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.334400    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:50.334581 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.334486    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs podName:18397989-832c-4e4d-b309-f09e6c5e0207 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:54.334452728 +0000 UTC m=+10.017912859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs") pod "network-metrics-daemon-hb6qs" (UID: "18397989-832c-4e4d-b309-f09e6c5e0207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:50.434768 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:50.434732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:50.434965 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:50.434786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:50.434965 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.434944    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:50.435079 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.435007    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret podName:bb1a056e-d8cc-4090-96ef-c2f53cfa99a8 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:54.434985983 +0000 UTC m=+10.118446118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret") pod "global-pull-secret-syncer-zrqsh" (UID: "bb1a056e-d8cc-4090-96ef-c2f53cfa99a8") : object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:50.435479 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.435323    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 07:04:50.435479 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.435352    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 07:04:50.435479 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.435367    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hmtjs for pod openshift-network-diagnostics/network-check-target-gtz5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:50.435479 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.435429    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs podName:34662fee-4240-4d04-8694-a55dc1510dc4 nodeName:}" failed. No retries permitted until 2026-09-08 07:04:54.435410678 +0000 UTC m=+10.118870812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmtjs" (UniqueName: "kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs") pod "network-check-target-gtz5m" (UID: "34662fee-4240-4d04-8694-a55dc1510dc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:50.937239 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:50.936853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:50.937239 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:50.936992    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:04:51.934693 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:51.934657    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:51.934876 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:51.934797    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:04:51.934876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:51.934657    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:51.935011 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:51.934899    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:04:52.934354 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:52.934277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:52.934807 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:52.934426    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:04:53.934861 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:53.934823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:53.935349 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:53.934961    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:04:53.935430 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:53.935388    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:53.935583 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:53.935554    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:04:54.371647 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:54.371606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:54.371820 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.371771    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:54.371894 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.371835    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs podName:18397989-832c-4e4d-b309-f09e6c5e0207 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:02.371815859 +0000 UTC m=+18.055275994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs") pod "network-metrics-daemon-hb6qs" (UID: "18397989-832c-4e4d-b309-f09e6c5e0207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:04:54.473035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:54.472991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:54.473240 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:54.473051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:54.473240 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.473212    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:54.473348 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.473276    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret podName:bb1a056e-d8cc-4090-96ef-c2f53cfa99a8 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:02.473255788 +0000 UTC m=+18.156715943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret") pod "global-pull-secret-syncer-zrqsh" (UID: "bb1a056e-d8cc-4090-96ef-c2f53cfa99a8") : object "kube-system"/"original-pull-secret" not registered
Sep 08 07:04:54.473410 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.473362    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 07:04:54.473410 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.473375    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 07:04:54.473410 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.473388    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hmtjs for pod openshift-network-diagnostics/network-check-target-gtz5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:54.473583 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.473422    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs podName:34662fee-4240-4d04-8694-a55dc1510dc4 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:02.473410778 +0000 UTC m=+18.156870915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmtjs" (UniqueName: "kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs") pod "network-check-target-gtz5m" (UID: "34662fee-4240-4d04-8694-a55dc1510dc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:04:54.935905 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:54.935870    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:54.936308 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:54.935998    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:04:55.934148 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:55.934111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:55.934348 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:55.934120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:55.934348 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:55.934249    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:04:55.934348 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:55.934334    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:04:56.934659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:56.934621    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:56.935117 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:56.934755    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:04:57.934659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:57.934624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:57.934856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:57.934666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:57.934856 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:57.934762    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:04:57.935222 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:57.934879    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:04:58.934394 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:58.934155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:04:58.934570 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:58.934532    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:04:59.934837 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:59.934801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:04:59.935273 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:59.934927    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:04:59.935273 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:04:59.934980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:04:59.935273 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:04:59.935088    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:00.934449 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:00.934415    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:00.934610 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:00.934559    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:01.934548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:01.934513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:01.934992 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:01.934553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:01.934992 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:01.934635    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:01.934992 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:01.934750    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:02.431167 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:02.431125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:02.431367 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.431260    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:05:02.431367 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.431327    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs podName:18397989-832c-4e4d-b309-f09e6c5e0207 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:18.431305974 +0000 UTC m=+34.114766157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs") pod "network-metrics-daemon-hb6qs" (UID: "18397989-832c-4e4d-b309-f09e6c5e0207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:05:02.532392 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:02.532352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:02.532566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:02.532405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:02.532566 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.532523    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 07:05:02.532566 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.532545    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 07:05:02.532566 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.532568    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 07:05:02.532736 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.532599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret podName:bb1a056e-d8cc-4090-96ef-c2f53cfa99a8 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:18.532582944 +0000 UTC m=+34.216043075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret") pod "global-pull-secret-syncer-zrqsh" (UID: "bb1a056e-d8cc-4090-96ef-c2f53cfa99a8") : object "kube-system"/"original-pull-secret" not registered
Sep 08 07:05:02.532736 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.532611    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hmtjs for pod openshift-network-diagnostics/network-check-target-gtz5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:05:02.532736 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.532663    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs podName:34662fee-4240-4d04-8694-a55dc1510dc4 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:18.532646806 +0000 UTC m=+34.216106952 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmtjs" (UniqueName: "kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs") pod "network-check-target-gtz5m" (UID: "34662fee-4240-4d04-8694-a55dc1510dc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:05:02.934551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:02.934517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:02.934993 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:02.934656    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:03.934794 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:03.934754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:03.935252 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:03.934812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:03.935252 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:03.934900    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:03.935252 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:03.935011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:04.935578 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:04.935114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:04.935578 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:04.935236    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:05.023686 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:05.023386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal" event={"ID":"901888addc1cc153e7772482f8ca35e4","Type":"ContainerStarted","Data":"a52b06c26e52a7b3d267afc3bd593b5ba2c0604eba3cf98d217c149e4ebcd3e2"}
Sep 08 07:05:05.031700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:05.031664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rlrr9" event={"ID":"821a0f41-2105-4003-b56f-17eec112c8bd","Type":"ContainerStarted","Data":"fa2edc72fb2c1ce5426a4fe9cda6ab29b84ba9b0f72abde2d082e5055e4f2449"}
Sep 08 07:05:05.037071 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:05.037021    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-109.ec2.internal" podStartSLOduration=19.037005669 podStartE2EDuration="19.037005669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:05.036677266 +0000 UTC m=+20.720137422" watchObservedRunningTime="2026-09-08 07:05:05.037005669 +0000 UTC m=+20.720465898"
Sep 08 07:05:05.045702 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:05.045672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qwkps" event={"ID":"8b7a2c1b-1105-437c-bdca-c0e6495ff408","Type":"ContainerStarted","Data":"1ab7a819cf7f54f98420f390f97c1999577dd9eb5c789291564d649d484990db"}
Sep 08 07:05:05.072829 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:05.072783    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rlrr9" podStartSLOduration=3.328580806 podStartE2EDuration="21.072765007s" totalImagesPullingTime="17.744184201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:05.055489388 +0000 UTC m=+20.738949542" watchObservedRunningTime="2026-09-08 07:05:05.072765007 +0000 UTC m=+20.756225201"
Sep 08 07:05:05.073148 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:05.073107    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qwkps" podStartSLOduration=3.338109188 podStartE2EDuration="21.073092765s" totalImagesPullingTime="17.734983577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:05.072591572 +0000 UTC m=+20.756051726" watchObservedRunningTime="2026-09-08 07:05:05.073092765 +0000 UTC m=+20.756552920"
Sep 08 07:05:05.934995 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:05.934822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:05.935164 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:05.934866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:05.935164 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:05.935066    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:05.935254 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:05.935157    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:06.049432 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.049335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lc7d6" event={"ID":"3e4a766e-393c-42b7-912b-5aabacc745f4","Type":"ContainerStarted","Data":"577201f7db84a4fc56824c61bc975bf809ca2a64de522975a91220636943e93b"}
Sep 08 07:05:06.051169 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.051139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5" event={"ID":"e1280dda-1e18-457e-b25c-b9bd2f5cd880","Type":"ContainerStarted","Data":"557f3af31515cc9a75b7824b0ed20a5bb4c6ade3bb350d439c80f5ed3ec57e74"}
Sep 08 07:05:06.052966 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.052930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6g8t9" event={"ID":"40e80d3a-bc71-4e0b-8616-f022edcc6fcb","Type":"ContainerStarted","Data":"dfe607b399ecd301e2fd1adac43b1b3c919390e075b438fcd719a4226dbb56f2"}
Sep 08 07:05:06.054962 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.054935    2580 generic.go:356] "Generic (PLEG): container finished" podID="4c2563eab4d47e90fa012e780aa92e92" containerID="2ed3e2ae56fdd44b6d94af47472ab19a4d1f1f6d5d0311af9d9dd6690f94e9d3" exitCode=0
Sep 08 07:05:06.055187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.055157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal" event={"ID":"4c2563eab4d47e90fa012e780aa92e92","Type":"ContainerDied","Data":"2ed3e2ae56fdd44b6d94af47472ab19a4d1f1f6d5d0311af9d9dd6690f94e9d3"}
Sep 08 07:05:06.056901 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.056871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v95nw" event={"ID":"d745d326-a791-4108-b6bd-59caff1b804b","Type":"ContainerStarted","Data":"55656b17e07262c3962c445f9de3ec9b086a41ba41e4ff9f1310bca0486c8adc"}
Sep 08 07:05:06.060336 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.060317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"fa49fb352040895e33e09428ba2fc18db44bd1a3480131b13f96e6a43f19907b"}
Sep 08 07:05:06.060416 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.060382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"f0898f4c869d471975294aa302f6aacb4114787823237c222cf6924e6d8071c4"}
Sep 08 07:05:06.060506 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.060420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"affb81b6dd3964f7d17ba51e44a8504a1c009a2e9d68266b7170d3dd905945dc"}
Sep 08 07:05:06.060506 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.060480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"875d3f322ce2bf347918203d8d15854d3c9c04c6b1218b0ea632ea5b5ecd8099"}
Sep 08 07:05:06.060594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.060512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"775cbbc2612df00dfe60216a7a77d0de4450a646af88da7b36f62fc955ec988d"}
Sep 08 07:05:06.060594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.060539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"b40f65bdf7ff551517941f94def17ddf108435f5bc2c1de961ee3724efd0330d"}
Sep 08 07:05:06.061798 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.061774    2580 generic.go:356] "Generic (PLEG): container finished" podID="37fef4ff-7a57-420d-ae7b-0dfd37943b3f" containerID="a2dde505e0624a5bfc0878aa65298846e7b9f59e5933c27f56a9de0d98bba340" exitCode=0
Sep 08 07:05:06.061877 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.061860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" event={"ID":"37fef4ff-7a57-420d-ae7b-0dfd37943b3f","Type":"ContainerDied","Data":"a2dde505e0624a5bfc0878aa65298846e7b9f59e5933c27f56a9de0d98bba340"}
Sep 08 07:05:06.064906 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.064615    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lc7d6" podStartSLOduration=7.157088088 podStartE2EDuration="21.064599234s" totalImagesPullingTime="13.907511146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:06.064404345 +0000 UTC m=+21.747864501" watchObservedRunningTime="2026-09-08 07:05:06.064599234 +0000 UTC m=+21.748059389"
Sep 08 07:05:06.095840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.095658    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v95nw" podStartSLOduration=7.202254582 podStartE2EDuration="21.095638785s" totalImagesPullingTime="13.893384203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:06.095170113 +0000 UTC m=+21.778630267" watchObservedRunningTime="2026-09-08 07:05:06.095638785 +0000 UTC m=+21.779098939"
Sep 08 07:05:06.122471 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.122404    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6g8t9" podStartSLOduration=4.393364475 podStartE2EDuration="22.122383915s" totalImagesPullingTime="17.72901944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:06.122042267 +0000 UTC m=+21.805502419" watchObservedRunningTime="2026-09-08 07:05:06.122383915 +0000 UTC m=+21.805844069"
Sep 08 07:05:06.196165 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.196138    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 07:05:06.672203 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.672152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:06.773135 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.773084    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:06.773315 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.773044    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T07:05:06.196160076Z","UUID":"c8ac0a3c-1e76-411f-b7c3-215e4364edbc","Handler":null,"Name":"","Endpoint":""}
Sep 08 07:05:06.774709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.774686    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 07:05:06.774831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.774723    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 07:05:06.934602 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:06.934517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:06.936659 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:06.936629    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:07.067703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.067661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nhxll" event={"ID":"f6b2b96b-9a18-4ba9-b565-d32aafb3cb07","Type":"ContainerStarted","Data":"b53c93b122e1d5789107f637735ad3f3c46bd879c1444a725f2185ebe21dd04a"}
Sep 08 07:05:07.070666 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.070629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5" event={"ID":"e1280dda-1e18-457e-b25c-b9bd2f5cd880","Type":"ContainerStarted","Data":"772355b14031f2dae39a550a061c5d14a5c608f516c8bf8180e001d3c0ba337f"}
Sep 08 07:05:07.074182 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.074124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal" event={"ID":"4c2563eab4d47e90fa012e780aa92e92","Type":"ContainerStarted","Data":"5771bee9d6c8ba16cd7202712b2634039cd09894b257fa222839c1e634a8c4e0"}
Sep 08 07:05:07.090438 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.090395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:07.098736 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.098662    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nhxll" podStartSLOduration=14.052433121 podStartE2EDuration="22.098639617s" totalImagesPullingTime="8.046206496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:07.083728788 +0000 UTC m=+22.767188942" watchObservedRunningTime="2026-09-08 07:05:07.098639617 +0000 UTC m=+22.782099771"
Sep 08 07:05:07.099376 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.099342    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-109.ec2.internal" podStartSLOduration=22.099328057 podStartE2EDuration="22.099328057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:07.098556322 +0000 UTC m=+22.782016500" watchObservedRunningTime="2026-09-08 07:05:07.099328057 +0000 UTC m=+22.782788212"
Sep 08 07:05:07.287292 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.287204    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:07.485433 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.485402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:07.586144 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.586110    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:07.741920 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.741874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:07.920565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.920533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:07.934583 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.934556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:07.934700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:07.934592    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:07.934700 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:07.934671    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:07.934815 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:07.934794    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:08.080643 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:08.080606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"bc97e1b26df4d97c785857ab19d19c4abd1b5f2bdf7ecc0ce70f5323ce88b83b"}
Sep 08 07:05:08.082960 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:08.082930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5" event={"ID":"e1280dda-1e18-457e-b25c-b9bd2f5cd880","Type":"ContainerStarted","Data":"7531bf7afdfd954c9c6059f5627a3dbf57e0b19dad98a6df8c62071ea6189b42"}
Sep 08 07:05:08.103219 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:08.103172    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjcf5" podStartSLOduration=3.488043733 podStartE2EDuration="23.103152821s" totalImagesPullingTime="19.615109088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:08.103138516 +0000 UTC m=+23.786598669" watchObservedRunningTime="2026-09-08 07:05:08.103152821 +0000 UTC m=+23.786612978"
Sep 08 07:05:08.690611 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:08.690573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:08.885504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:08.885472    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:08.934725 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:08.934693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:08.934892 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:08.934856    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:09.270474 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:09.270436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:09.320810 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:09.320777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:09.380959 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:09.380933    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:05:09.381532 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:09.381510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:05:09.500443 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:09.500412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:09.886877 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:09.886790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:09.934090 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:09.934059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:09.934267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:09.934059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:09.934267 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:09.934187    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:09.934267 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:09.934253    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:10.489940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:10.489911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:10.934449 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:10.934425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:10.934602 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:10.934577    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:11.094027 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.093991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" event={"ID":"0c075acc-7e18-42fb-8778-e05d68b02569","Type":"ContainerStarted","Data":"603bfb6df5258bb89ce3315a183e62c00555de0b73ba8b4d37c6706e48363935"}
Sep 08 07:05:11.094414 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.094307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:05:11.110683 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.110514    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:05:11.124046 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.123958    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh" podStartSLOduration=8.264597752 podStartE2EDuration="26.123942885s" totalImagesPullingTime="17.859345133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:11.123563131 +0000 UTC m=+26.807023283" watchObservedRunningTime="2026-09-08 07:05:11.123942885 +0000 UTC m=+26.807403015"
Sep 08 07:05:11.131693 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.131662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:11.288801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.288776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:11.692516 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.692425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:11.890336 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.890304    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:11.934491 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.934448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:11.934631 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:11.934564    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:11.934631 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:11.934595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:11.934699 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:11.934650    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:12.097924 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.097889    2580 generic.go:356] "Generic (PLEG): container finished" podID="37fef4ff-7a57-420d-ae7b-0dfd37943b3f" containerID="ca63bff7a686d05302a228e2370b3f81f37885ec39873be4e794d1374f98da1d" exitCode=0
Sep 08 07:05:12.098134 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.097939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" event={"ID":"37fef4ff-7a57-420d-ae7b-0dfd37943b3f","Type":"ContainerDied","Data":"ca63bff7a686d05302a228e2370b3f81f37885ec39873be4e794d1374f98da1d"}
Sep 08 07:05:12.098373 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.098355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:05:12.098477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.098417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:05:12.113029 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.113005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:05:12.190684 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.190662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:05:12.190833 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.190753    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 07:05:12.191120 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.191104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6g8t9"
Sep 08 07:05:12.290265 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.290236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:12.690472 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.690427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:12.934528 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:12.934504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:12.935020 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:12.934637    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:13.342017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:13.341990    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:13.493433 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:13.493352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:13.686052 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:13.686023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:13.934609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:13.934583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:13.934977 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:13.934582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:13.934977 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:13.934677    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:13.934977 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:13.934744    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:14.107437 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:14.107406    2580 generic.go:356] "Generic (PLEG): container finished" podID="37fef4ff-7a57-420d-ae7b-0dfd37943b3f" containerID="f0593f7366fd365d8d256b45be8626e445a11f68e576fca99434379b226a7ee8" exitCode=0
Sep 08 07:05:14.107595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:14.107488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" event={"ID":"37fef4ff-7a57-420d-ae7b-0dfd37943b3f","Type":"ContainerDied","Data":"f0593f7366fd365d8d256b45be8626e445a11f68e576fca99434379b226a7ee8"}
Sep 08 07:05:14.935085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:14.935057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:14.935613 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:14.935164    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:15.115448 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:15.115412    2580 generic.go:356] "Generic (PLEG): container finished" podID="37fef4ff-7a57-420d-ae7b-0dfd37943b3f" containerID="ee2ae1b848363c3829592abc909ba44d8736a885bc65cb4aed3f739a92ba1fcb" exitCode=0
Sep 08 07:05:15.115628 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:15.115490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" event={"ID":"37fef4ff-7a57-420d-ae7b-0dfd37943b3f","Type":"ContainerDied","Data":"ee2ae1b848363c3829592abc909ba44d8736a885bc65cb4aed3f739a92ba1fcb"}
Sep 08 07:05:15.934032 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:15.933994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:15.934032 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:15.934031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:15.934266 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:15.934104    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:15.934266 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:15.934193    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:16.935075 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:16.935036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:16.935573 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:16.935175    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:17.934762 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:17.934732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:17.934943 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:17.934850    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:17.934943 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:17.934904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:17.935050 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:17.935015    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:18.175188 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.175157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:18.275692 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.275610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:18.334291 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.334259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:18.457229 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.457196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:18.457383 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.457335    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:05:18.457439 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.457387    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs podName:18397989-832c-4e4d-b309-f09e6c5e0207 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:50.457373723 +0000 UTC m=+66.140833855 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs") pod "network-metrics-daemon-hb6qs" (UID: "18397989-832c-4e4d-b309-f09e6c5e0207") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 07:05:18.469812 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.469782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:18.558391 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.558299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:18.558391 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.558346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:18.558624 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.558453    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 07:05:18.558624 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.558487    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 07:05:18.558624 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.558515    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 07:05:18.558624 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.558528    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret podName:bb1a056e-d8cc-4090-96ef-c2f53cfa99a8 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:50.558507992 +0000 UTC m=+66.241968127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret") pod "global-pull-secret-syncer-zrqsh" (UID: "bb1a056e-d8cc-4090-96ef-c2f53cfa99a8") : object "kube-system"/"original-pull-secret" not registered
Sep 08 07:05:18.558624 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.558529    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hmtjs for pod openshift-network-diagnostics/network-check-target-gtz5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:05:18.558624 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.558573    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs podName:34662fee-4240-4d04-8694-a55dc1510dc4 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:50.558563269 +0000 UTC m=+66.242023400 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmtjs" (UniqueName: "kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs") pod "network-check-target-gtz5m" (UID: "34662fee-4240-4d04-8694-a55dc1510dc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 07:05:18.853237 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.853194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:18.934291 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:18.934260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:18.934486 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:18.934390    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:19.049803 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:19.049753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:19.151530 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:19.151506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:19.255617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:19.255591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:19.628618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:19.628586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:19.728971 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:19.728938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:19.855477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:19.855438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:19.934357 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:19.934287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:19.934357 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:19.934326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:19.934524 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:19.934396    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:19.934524 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:19.934516    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:20.070019 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.069978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:20.130131 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.130098    2580 generic.go:356] "Generic (PLEG): container finished" podID="37fef4ff-7a57-420d-ae7b-0dfd37943b3f" containerID="117a189548ebe3ffc8a92a35db2746d0e0a2c479c829e23969aa7df8e243e928" exitCode=0
Sep 08 07:05:20.130308 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.130141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" event={"ID":"37fef4ff-7a57-420d-ae7b-0dfd37943b3f","Type":"ContainerDied","Data":"117a189548ebe3ffc8a92a35db2746d0e0a2c479c829e23969aa7df8e243e928"}
Sep 08 07:05:20.273709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.273679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:20.352273 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.352249    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:20.452938 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.452913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:20.552842 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.552634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:20.650680 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.650645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:20.934247 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.934213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:20.934415 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:20.934334    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:20.986017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.985981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hb6qs"]
Sep 08 07:05:20.988876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.988851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zrqsh"]
Sep 08 07:05:20.989162 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.989144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:20.989362 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:20.989337    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:20.989637 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.989616    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gtz5m"]
Sep 08 07:05:20.989708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:20.989691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:20.989800 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:20.989781    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:21.080320 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:21.080290    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:21.135922 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:21.135890    2580 generic.go:356] "Generic (PLEG): container finished" podID="37fef4ff-7a57-420d-ae7b-0dfd37943b3f" containerID="80e7dd5e11b2ab26d76b8c26587644718e16608925e25103154f7061638eaad4" exitCode=0
Sep 08 07:05:21.136173 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:21.135969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" event={"ID":"37fef4ff-7a57-420d-ae7b-0dfd37943b3f","Type":"ContainerDied","Data":"80e7dd5e11b2ab26d76b8c26587644718e16608925e25103154f7061638eaad4"}
Sep 08 07:05:21.136173 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:21.136052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:21.136348 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:21.136318    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:21.136559 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:21.136540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:22.142046 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.142020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" event={"ID":"37fef4ff-7a57-420d-ae7b-0dfd37943b3f","Type":"ContainerStarted","Data":"d6050da8d095acb8c680442287e66acc90002b8d70653d4d8d04fc8b9f0af413"}
Sep 08 07:05:22.162921 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.162881    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8nvwv" podStartSLOduration=17.225534595 podStartE2EDuration="38.162865897s" totalImagesPullingTime="20.937331302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:22.161711294 +0000 UTC m=+37.845171460" watchObservedRunningTime="2026-09-08 07:05:22.162865897 +0000 UTC m=+37.846326050"
Sep 08 07:05:22.234041 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.234013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:22.333606 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.333579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:22.437325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.437247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:22.491347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.491319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:22.841288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.841261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:22.896124 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.896097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:22.934304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.934282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:22.934413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.934282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:22.934413 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:22.934385    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:22.934501 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:22.934412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:22.934534 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:22.934509    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:22.934619 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:22.934603    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:24.640279 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:24.640080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:24.742802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:24.742775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:24.802556 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:24.802523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:24.935454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:24.935372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:24.935454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:24.935443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:24.935670 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:24.935478    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6qs" podUID="18397989-832c-4e4d-b309-f09e6c5e0207"
Sep 08 07:05:24.935670 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:24.935551    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrqsh" podUID="bb1a056e-d8cc-4090-96ef-c2f53cfa99a8"
Sep 08 07:05:24.935670 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:24.935560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:24.935670 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:24.935621    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gtz5m" podUID="34662fee-4240-4d04-8694-a55dc1510dc4"
Sep 08 07:05:25.061418 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.061388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:25.139162 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.139133    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:25.242910 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.242827    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:25.342753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.342716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:25.442621 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.442593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:25.525687 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.525630    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-109.ec2.internal" event="NodeReady"
Sep 08 07:05:25.525782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.525747    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 07:05:25.566817 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.566785    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6t76l"]
Sep 08 07:05:25.593059 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.593030    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rczlb"]
Sep 08 07:05:25.593236 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.593185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6t76l"
Sep 08 07:05:25.595285 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.595260    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 07:05:25.595446 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.595305    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 07:05:25.595654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.595636    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qkwpw\""
Sep 08 07:05:25.595862 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.595847    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 07:05:25.608982 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.608959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6t76l"]
Sep 08 07:05:25.609096 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.608996    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q8rpz"]
Sep 08 07:05:25.609096 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.609083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.611141 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.611122    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lv5z5\""
Sep 08 07:05:25.611246 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.611190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 07:05:25.611246 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.611217    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 07:05:25.611246 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.611194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 07:05:25.611404 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.611219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 07:05:25.631015 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.630991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rczlb"]
Sep 08 07:05:25.631129 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.631028    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q8rpz"]
Sep 08 07:05:25.631129 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.631109    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.633194 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.633175    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 07:05:25.633194 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.633183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 07:05:25.633341 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.633183    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mmqzm\""
Sep 08 07:05:25.692646 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.692615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:25.709636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.709610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c45c7a-3c98-48b3-aba0-e7da22438105-cert\") pod \"ingress-canary-6t76l\" (UID: \"e2c45c7a-3c98-48b3-aba0-e7da22438105\") " pod="openshift-ingress-canary/ingress-canary-6t76l"
Sep 08 07:05:25.709711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.709655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0347d928-62cf-4ad9-8bdf-615920a4ee9e-data-volume\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.709711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.709673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0347d928-62cf-4ad9-8bdf-615920a4ee9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.709789 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.709720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnj9z\" (UniqueName: \"kubernetes.io/projected/0347d928-62cf-4ad9-8bdf-615920a4ee9e-kube-api-access-pnj9z\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.709789 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.709770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jgj\" (UniqueName: \"kubernetes.io/projected/e2c45c7a-3c98-48b3-aba0-e7da22438105-kube-api-access-b4jgj\") pod \"ingress-canary-6t76l\" (UID: \"e2c45c7a-3c98-48b3-aba0-e7da22438105\") " pod="openshift-ingress-canary/ingress-canary-6t76l"
Sep 08 07:05:25.709851 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.709793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0347d928-62cf-4ad9-8bdf-615920a4ee9e-crio-socket\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.709851 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.709808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0347d928-62cf-4ad9-8bdf-615920a4ee9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.780333 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.780267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:25.810425 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c45c7a-3c98-48b3-aba0-e7da22438105-cert\") pod \"ingress-canary-6t76l\" (UID: \"e2c45c7a-3c98-48b3-aba0-e7da22438105\") " pod="openshift-ingress-canary/ingress-canary-6t76l"
Sep 08 07:05:25.810565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbjg\" (UniqueName: \"kubernetes.io/projected/e7f79343-7c0b-4029-a54e-d73b1b5e7007-kube-api-access-tgbjg\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.810565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0347d928-62cf-4ad9-8bdf-615920a4ee9e-data-volume\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.810565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0347d928-62cf-4ad9-8bdf-615920a4ee9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.810565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f79343-7c0b-4029-a54e-d73b1b5e7007-config-volume\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.810565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnj9z\" (UniqueName: \"kubernetes.io/projected/0347d928-62cf-4ad9-8bdf-615920a4ee9e-kube-api-access-pnj9z\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.810565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7f79343-7c0b-4029-a54e-d73b1b5e7007-metrics-tls\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.810854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jgj\" (UniqueName: \"kubernetes.io/projected/e2c45c7a-3c98-48b3-aba0-e7da22438105-kube-api-access-b4jgj\") pod \"ingress-canary-6t76l\" (UID: \"e2c45c7a-3c98-48b3-aba0-e7da22438105\") " pod="openshift-ingress-canary/ingress-canary-6t76l"
Sep 08 07:05:25.810854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0347d928-62cf-4ad9-8bdf-615920a4ee9e-crio-socket\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.810854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0347d928-62cf-4ad9-8bdf-615920a4ee9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.810854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f79343-7c0b-4029-a54e-d73b1b5e7007-tmp-dir\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.810854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0347d928-62cf-4ad9-8bdf-615920a4ee9e-data-volume\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.811016 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.810867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0347d928-62cf-4ad9-8bdf-615920a4ee9e-crio-socket\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.811228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.811199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0347d928-62cf-4ad9-8bdf-615920a4ee9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.814339 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.814309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0347d928-62cf-4ad9-8bdf-615920a4ee9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.814416 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.814351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c45c7a-3c98-48b3-aba0-e7da22438105-cert\") pod \"ingress-canary-6t76l\" (UID: \"e2c45c7a-3c98-48b3-aba0-e7da22438105\") " pod="openshift-ingress-canary/ingress-canary-6t76l"
Sep 08 07:05:25.817515 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.817488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jgj\" (UniqueName: \"kubernetes.io/projected/e2c45c7a-3c98-48b3-aba0-e7da22438105-kube-api-access-b4jgj\") pod \"ingress-canary-6t76l\" (UID: \"e2c45c7a-3c98-48b3-aba0-e7da22438105\") " pod="openshift-ingress-canary/ingress-canary-6t76l"
Sep 08 07:05:25.817745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.817724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnj9z\" (UniqueName: \"kubernetes.io/projected/0347d928-62cf-4ad9-8bdf-615920a4ee9e-kube-api-access-pnj9z\") pod \"insights-runtime-extractor-rczlb\" (UID: \"0347d928-62cf-4ad9-8bdf-615920a4ee9e\") " pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.903228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.903188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6t76l"
Sep 08 07:05:25.911510 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.911482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f79343-7c0b-4029-a54e-d73b1b5e7007-tmp-dir\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.911595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.911526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbjg\" (UniqueName: \"kubernetes.io/projected/e7f79343-7c0b-4029-a54e-d73b1b5e7007-kube-api-access-tgbjg\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.911595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.911559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f79343-7c0b-4029-a54e-d73b1b5e7007-config-volume\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.911595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.911578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7f79343-7c0b-4029-a54e-d73b1b5e7007-metrics-tls\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.911807 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.911787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f79343-7c0b-4029-a54e-d73b1b5e7007-tmp-dir\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.912138 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.912117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f79343-7c0b-4029-a54e-d73b1b5e7007-config-volume\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.913780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.913762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7f79343-7c0b-4029-a54e-d73b1b5e7007-metrics-tls\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.919006 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.918983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbjg\" (UniqueName: \"kubernetes.io/projected/e7f79343-7c0b-4029-a54e-d73b1b5e7007-kube-api-access-tgbjg\") pod \"dns-default-q8rpz\" (UID: \"e7f79343-7c0b-4029-a54e-d73b1b5e7007\") " pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:25.919890 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.919872    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rczlb"
Sep 08 07:05:25.938621 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:25.938583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:26.218908 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.218878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6t76l"]
Sep 08 07:05:26.226514 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:26.226489    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c45c7a_3c98_48b3_aba0_e7da22438105.slice/crio-50c64f9ee6f475d7e6d130e0d97831e2468ae50e022441dbda69c467dd6a1841 WatchSource:0}: Error finding container 50c64f9ee6f475d7e6d130e0d97831e2468ae50e022441dbda69c467dd6a1841: Status 404 returned error can't find the container with id 50c64f9ee6f475d7e6d130e0d97831e2468ae50e022441dbda69c467dd6a1841
Sep 08 07:05:26.234267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.234244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rczlb"]
Sep 08 07:05:26.237308 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:26.237284    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0347d928_62cf_4ad9_8bdf_615920a4ee9e.slice/crio-9c9752b870278dae0a5b1c61d1c321f7328e574268a8f9140af4152c90746ec9 WatchSource:0}: Error finding container 9c9752b870278dae0a5b1c61d1c321f7328e574268a8f9140af4152c90746ec9: Status 404 returned error can't find the container with id 9c9752b870278dae0a5b1c61d1c321f7328e574268a8f9140af4152c90746ec9
Sep 08 07:05:26.253068 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.253042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q8rpz"]
Sep 08 07:05:26.257894 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:26.257869    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f79343_7c0b_4029_a54e_d73b1b5e7007.slice/crio-20e3cb6e1d0997aabf48368a725e8d9a6d63f6ef84b6276b925850831d2b6bd5 WatchSource:0}: Error finding container 20e3cb6e1d0997aabf48368a725e8d9a6d63f6ef84b6276b925850831d2b6bd5: Status 404 returned error can't find the container with id 20e3cb6e1d0997aabf48368a725e8d9a6d63f6ef84b6276b925850831d2b6bd5
Sep 08 07:05:26.934359 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.934157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:26.935209 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.934157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:26.935209 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.934157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:26.937304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.937284    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 07:05:26.937559 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.937541    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ldq7w\""
Sep 08 07:05:26.937690 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.937563    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7mcm7\""
Sep 08 07:05:26.937865 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.937826    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 07:05:26.938042 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.938029    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 07:05:26.938269 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:26.938250    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 07:05:27.155599 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.155562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rczlb" event={"ID":"0347d928-62cf-4ad9-8bdf-615920a4ee9e","Type":"ContainerStarted","Data":"3cfe0401c6209d4b213782750ae71fc55aff258d78ccf43604a4354bd96519be"}
Sep 08 07:05:27.155782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.155630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rczlb" event={"ID":"0347d928-62cf-4ad9-8bdf-615920a4ee9e","Type":"ContainerStarted","Data":"9c9752b870278dae0a5b1c61d1c321f7328e574268a8f9140af4152c90746ec9"}
Sep 08 07:05:27.156887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.156849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8rpz" event={"ID":"e7f79343-7c0b-4029-a54e-d73b1b5e7007","Type":"ContainerStarted","Data":"20e3cb6e1d0997aabf48368a725e8d9a6d63f6ef84b6276b925850831d2b6bd5"}
Sep 08 07:05:27.158065 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.158039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6t76l" event={"ID":"e2c45c7a-3c98-48b3-aba0-e7da22438105","Type":"ContainerStarted","Data":"50c64f9ee6f475d7e6d130e0d97831e2468ae50e022441dbda69c467dd6a1841"}
Sep 08 07:05:27.611960 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.611922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:27.701285 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.701210    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64f959d977-kwf6h"]
Sep 08 07:05:27.712569 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.712539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.715074 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.715045    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 07:05:27.715074 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.715045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 07:05:27.715301 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.715047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 07:05:27.715894 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.715847    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mxlnj\""
Sep 08 07:05:27.716093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.716004    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 07:05:27.716093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.716083    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 07:05:27.716561 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.716400    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 07:05:27.716858 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.716721    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 07:05:27.717730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.717707    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f959d977-kwf6h"]
Sep 08 07:05:27.826024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.825833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-oauth-config\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.826024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.825895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-serving-cert\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.826024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.825926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gztbs\" (UniqueName: \"kubernetes.io/projected/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-kube-api-access-gztbs\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.826366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.826043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-oauth-serving-cert\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.826366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.826095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-service-ca\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.826366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.826122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-config\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.926524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.926492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-oauth-config\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.926708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.926547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-serving-cert\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.926708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.926570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gztbs\" (UniqueName: \"kubernetes.io/projected/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-kube-api-access-gztbs\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.926708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.926643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-oauth-serving-cert\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.926708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.926661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-service-ca\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.926708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.926681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-config\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.929714 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.928450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-service-ca\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.929714 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.928357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-config\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.929714 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.929086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-oauth-serving-cert\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.932788 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.932763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-oauth-config\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.932914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.932852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-serving-cert\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:27.937888 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:27.937861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gztbs\" (UniqueName: \"kubernetes.io/projected/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-kube-api-access-gztbs\") pod \"console-64f959d977-kwf6h\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") " pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:28.023687 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:28.023590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:28.162972 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:28.162927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rczlb" event={"ID":"0347d928-62cf-4ad9-8bdf-615920a4ee9e","Type":"ContainerStarted","Data":"5eae3b856fbb75976b68df26da696e3cd56c6ed88c79bb994fff60654417f047"}
Sep 08 07:05:30.125977 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:30.125685    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f959d977-kwf6h"]
Sep 08 07:05:30.169917 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:30.169883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8rpz" event={"ID":"e7f79343-7c0b-4029-a54e-d73b1b5e7007","Type":"ContainerStarted","Data":"6fa999e305f50d5c573a3a5ec69b47280c8ed012aa8b56f646fc45f1f08a2dc8"}
Sep 08 07:05:30.170080 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:30.169942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8rpz" event={"ID":"e7f79343-7c0b-4029-a54e-d73b1b5e7007","Type":"ContainerStarted","Data":"02a51a1dba64f4baf15bf5115e7d3a49de0cc9b6ebcb348664809d92e6c22a87"}
Sep 08 07:05:30.170080 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:30.169981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:30.171526 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:30.171494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6t76l" event={"ID":"e2c45c7a-3c98-48b3-aba0-e7da22438105","Type":"ContainerStarted","Data":"6aafedbaf7ffe2f069c6cf5f051f586122e0584c61085616b757622a72330918"}
Sep 08 07:05:30.184507 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:30.184446    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dc7fb9_c7a9_42a1_98ac_6a08557c6514.slice/crio-ae9d34bfebf2747f4d6c61248626a34e9a68303d81a4571604cea125b2ed4160 WatchSource:0}: Error finding container ae9d34bfebf2747f4d6c61248626a34e9a68303d81a4571604cea125b2ed4160: Status 404 returned error can't find the container with id ae9d34bfebf2747f4d6c61248626a34e9a68303d81a4571604cea125b2ed4160
Sep 08 07:05:30.186028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:30.185978    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q8rpz" podStartSLOduration=1.587131047 podStartE2EDuration="5.185958203s" totalImagesPullingTime="3.598827156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:30.185209381 +0000 UTC m=+45.868669534" watchObservedRunningTime="2026-09-08 07:05:30.185958203 +0000 UTC m=+45.869418357"
Sep 08 07:05:30.200107 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:30.200067    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6t76l" podStartSLOduration=1.5682626069999999 podStartE2EDuration="5.200039615s" totalImagesPullingTime="3.631777008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:30.199089661 +0000 UTC m=+45.882549811" watchObservedRunningTime="2026-09-08 07:05:30.200039615 +0000 UTC m=+45.883499799"
Sep 08 07:05:31.179821 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:31.179779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rczlb" event={"ID":"0347d928-62cf-4ad9-8bdf-615920a4ee9e","Type":"ContainerStarted","Data":"a3c19a1c2b1279882ca9ab5357d11d928582fa7fba7c087e56c2e8238490a0f2"}
Sep 08 07:05:31.181333 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:31.181305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f959d977-kwf6h" event={"ID":"66dc7fb9-c7a9-42a1-98ac-6a08557c6514","Type":"ContainerStarted","Data":"ae9d34bfebf2747f4d6c61248626a34e9a68303d81a4571604cea125b2ed4160"}
Sep 08 07:05:31.198012 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:31.197949    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rczlb" podStartSLOduration=2.461527975 podStartE2EDuration="6.197930002s" totalImagesPullingTime="3.736402027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:05:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:31.196091861 +0000 UTC m=+46.879552013" watchObservedRunningTime="2026-09-08 07:05:31.197930002 +0000 UTC m=+46.881390151"
Sep 08 07:05:33.083257 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:33.083222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:34.194039 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.194006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f959d977-kwf6h" event={"ID":"66dc7fb9-c7a9-42a1-98ac-6a08557c6514","Type":"ContainerStarted","Data":"c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94"}
Sep 08 07:05:34.217193 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.217144    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64f959d977-kwf6h" podStartSLOduration=3.923863104 podStartE2EDuration="7.217129475s" totalImagesPullingTime="3.293266371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:34.216224225 +0000 UTC m=+49.899684380" watchObservedRunningTime="2026-09-08 07:05:34.217129475 +0000 UTC m=+49.900589625"
Sep 08 07:05:34.646913 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.646743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:34.747867 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.747837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:34.836801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.836770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:34.839496 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.839476    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-l5gks"]
Sep 08 07:05:34.859782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.859757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-l5gks"]
Sep 08 07:05:34.859917 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.859846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.861930 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.861912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 08 07:05:34.862580 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.862554    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 08 07:05:34.862690 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.862563    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 07:05:34.865883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.863187    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xc4vv\""
Sep 08 07:05:34.865883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.864262    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 07:05:34.866235 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.866216    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 07:05:34.881377 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.881354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.881475 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.881403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.881475 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.881425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f496c7-6c94-4477-895f-90f5f2ff589d-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.881547 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.881474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhnq4\" (UniqueName: \"kubernetes.io/projected/f1f496c7-6c94-4477-895f-90f5f2ff589d-kube-api-access-zhnq4\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.897388 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.897319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:34.982739 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.982703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.982918 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.982771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.982918 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.982797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f496c7-6c94-4477-895f-90f5f2ff589d-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.982918 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.982814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhnq4\" (UniqueName: \"kubernetes.io/projected/f1f496c7-6c94-4477-895f-90f5f2ff589d-kube-api-access-zhnq4\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.982918 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:34.982862    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 08 07:05:34.983064 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:34.982935    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-tls podName:f1f496c7-6c94-4477-895f-90f5f2ff589d nodeName:}" failed. No retries permitted until 2026-09-08 07:05:35.482917974 +0000 UTC m=+51.166378109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-l5gks" (UID: "f1f496c7-6c94-4477-895f-90f5f2ff589d") : secret "prometheus-operator-tls" not found
Sep 08 07:05:34.983674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.983654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f496c7-6c94-4477-895f-90f5f2ff589d-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.987026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.987008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:34.990123 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:34.990103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhnq4\" (UniqueName: \"kubernetes.io/projected/f1f496c7-6c94-4477-895f-90f5f2ff589d-kube-api-access-zhnq4\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:35.329390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:35.329354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:35.486470 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:35.486409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:35.488753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:35.488727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f496c7-6c94-4477-895f-90f5f2ff589d-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-l5gks\" (UID: \"f1f496c7-6c94-4477-895f-90f5f2ff589d\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:35.771148 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:35.771069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks"
Sep 08 07:05:36.038802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:36.038721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-l5gks"]
Sep 08 07:05:36.042417 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:36.042390    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f496c7_6c94_4477_895f_90f5f2ff589d.slice/crio-0d7e237cf0b44e7619439177830ee2b3d671c4aa5b47effbee09102a1f7e884a WatchSource:0}: Error finding container 0d7e237cf0b44e7619439177830ee2b3d671c4aa5b47effbee09102a1f7e884a: Status 404 returned error can't find the container with id 0d7e237cf0b44e7619439177830ee2b3d671c4aa5b47effbee09102a1f7e884a
Sep 08 07:05:36.202276 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:36.202238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks" event={"ID":"f1f496c7-6c94-4477-895f-90f5f2ff589d","Type":"ContainerStarted","Data":"0d7e237cf0b44e7619439177830ee2b3d671c4aa5b47effbee09102a1f7e884a"}
Sep 08 07:05:36.479416 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:36.479383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:37.502977 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:37.502762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:37.602487 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:37.602440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:38.024034 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.023980    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:38.024222 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.024063    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:05:38.025442 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.025417    2580 patch_prober.go:28] interesting pod/console-64f959d977-kwf6h container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Sep 08 07:05:38.025534 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.025483    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-64f959d977-kwf6h" podUID="66dc7fb9-c7a9-42a1-98ac-6a08557c6514" containerName="console" probeResult="failure" output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused"
Sep 08 07:05:38.079740 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.079707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:38.140748 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.140715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:38.210118 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.210084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks" event={"ID":"f1f496c7-6c94-4477-895f-90f5f2ff589d","Type":"ContainerStarted","Data":"7ab753095edbb8f4ef220b07de48644d943dc142b731d21423dfee5b767ab357"}
Sep 08 07:05:38.210275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.210146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks" event={"ID":"f1f496c7-6c94-4477-895f-90f5f2ff589d","Type":"ContainerStarted","Data":"4e44aae82002f66afda0efacdb98034957295e6c06d02d4df6e81ff33ee6158e"}
Sep 08 07:05:38.226228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:38.226174    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-l5gks" podStartSLOduration=3.023858568 podStartE2EDuration="4.226152558s" totalImagesPullingTime="1.20229399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:38.22543834 +0000 UTC m=+53.908898492" watchObservedRunningTime="2026-09-08 07:05:38.226152558 +0000 UTC m=+53.909612713"
Sep 08 07:05:39.945441 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:39.945408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:40.163414 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.163382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:40.183931 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.183902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q8rpz"
Sep 08 07:05:40.243491 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.243399    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vm878"]
Sep 08 07:05:40.248800 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.248765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.250943 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.250916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 07:05:40.251091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.250976    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 07:05:40.251091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.251016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vfkd4\""
Sep 08 07:05:40.251091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.251025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 07:05:40.256629 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.256607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:40.318124 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.318090    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:40.324208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-tls\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.324340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.324340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-metrics-client-ca\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.324340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-root\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.324454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-textfile\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.324454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-sys\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.324454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-wtmp\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.324454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.324601 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.324521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd62v\" (UniqueName: \"kubernetes.io/projected/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-kube-api-access-wd62v\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.419369 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.419334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:40.425427 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425519 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd62v\" (UniqueName: \"kubernetes.io/projected/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-kube-api-access-wd62v\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425519 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-tls\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425519 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-metrics-client-ca\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-root\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-textfile\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-sys\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-wtmp\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-root\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-sys\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.425956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.425925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-wtmp\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.426058 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.426009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-textfile\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.426111 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.426094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.426158 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.426124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-metrics-client-ca\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.428901 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.428883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.428987 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.428922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-node-exporter-tls\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.432911 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.432879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd62v\" (UniqueName: \"kubernetes.io/projected/fc4e3f3b-6bd3-4752-82e3-a373c44729b5-kube-api-access-wd62v\") pod \"node-exporter-vm878\" (UID: \"fc4e3f3b-6bd3-4752-82e3-a373c44729b5\") " pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.521728 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.521648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:40.558272 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.558242    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vm878"
Sep 08 07:05:40.571007 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:40.570973    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4e3f3b_6bd3_4752_82e3_a373c44729b5.slice/crio-206725824c99a5c755edcb2ade2a854ff87186cf52743390a11c569212e3fe4b WatchSource:0}: Error finding container 206725824c99a5c755edcb2ade2a854ff87186cf52743390a11c569212e3fe4b: Status 404 returned error can't find the container with id 206725824c99a5c755edcb2ade2a854ff87186cf52743390a11c569212e3fe4b
Sep 08 07:05:40.620541 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.620508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:40.720363 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:40.720329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:41.130406 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.130318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:41.221432 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.221398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vm878" event={"ID":"fc4e3f3b-6bd3-4752-82e3-a373c44729b5","Type":"ContainerStarted","Data":"206725824c99a5c755edcb2ade2a854ff87186cf52743390a11c569212e3fe4b"}
Sep 08 07:05:41.229534 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.229501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:41.278958 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.278928    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 07:05:41.299850 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.299219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 07:05:41.299850 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.299366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.301420 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.301397    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 07:05:41.301553 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.301451    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 07:05:41.301553 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.301522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 07:05:41.301767 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.301742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 07:05:41.302062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.301881    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 07:05:41.302062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.301892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 07:05:41.302236 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.302117    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 07:05:41.302236 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.302205    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 07:05:41.302368 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.302340    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4rknp\""
Sep 08 07:05:41.302478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.302425    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 07:05:41.305609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.305587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:41.332881 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.332843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.332881 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.332880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4r7b\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-kube-api-access-m4r7b\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.332904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.332924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.332965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-web-config\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.333009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.333035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-out\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.333051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.333072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.333119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-volume\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.333165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.333183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.333314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.333224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-tls-assets\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433655 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-out\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433655 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433655 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-volume\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-tls-assets\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4r7b\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-kube-api-access-m4r7b\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.433928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.434259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.434259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-web-config\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.434259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.433983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.434259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.434116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.434674 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:41.434538    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 08 07:05:41.434674 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:41.434635    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls podName:72124c50-9bed-4bcf-adf4-41e6f33d5720 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:41.93461207 +0000 UTC m=+57.618072220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720") : secret "alertmanager-main-tls" not found
Sep 08 07:05:41.434945 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:41.434925    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle podName:72124c50-9bed-4bcf-adf4-41e6f33d5720 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:41.934908119 +0000 UTC m=+57.618368251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720") : configmap references non-existent config key: ca-bundle.crt
Sep 08 07:05:41.435035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.435010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.436874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.436849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-volume\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.436985 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.436849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.437138 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.437088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-out\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.437967 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.437652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-tls-assets\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.437967 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.437677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.437967 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.437913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.438133 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.438001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.438498 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.438449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-web-config\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.442571 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.442551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4r7b\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-kube-api-access-m4r7b\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.528753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.528704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:41.628716 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.628684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:41.938050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.938009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.938050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.938056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:41.938254 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:41.938176    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 08 07:05:41.938289 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:05:41.938259    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls podName:72124c50-9bed-4bcf-adf4-41e6f33d5720 nodeName:}" failed. No retries permitted until 2026-09-08 07:05:42.938235489 +0000 UTC m=+58.621695634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720") : secret "alertmanager-main-tls" not found
Sep 08 07:05:41.939191 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:41.939167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:42.230001 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:42.229916    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc4e3f3b-6bd3-4752-82e3-a373c44729b5" containerID="1552e8b8d1483aee86d9556b927f969ee8220f518a900921b1e89e6ef314e25f" exitCode=0
Sep 08 07:05:42.230001 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:42.229973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vm878" event={"ID":"fc4e3f3b-6bd3-4752-82e3-a373c44729b5","Type":"ContainerDied","Data":"1552e8b8d1483aee86d9556b927f969ee8220f518a900921b1e89e6ef314e25f"}
Sep 08 07:05:42.369789 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:42.369754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:42.947636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:42.947598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:42.949951 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:42.949919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:43.111733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.111700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:05:43.151845 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.151807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:43.201239 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.201163    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"]
Sep 08 07:05:43.203677 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.203661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.205832 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.205806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 08 07:05:43.205832 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.205806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 08 07:05:43.206076 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.206059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 08 07:05:43.206119 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.206090    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2kwww\""
Sep 08 07:05:43.206166 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.206089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 08 07:05:43.206400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.206379    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 08 07:05:43.206504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.206383    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7glj0no2k40l0\""
Sep 08 07:05:43.214671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.214653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"]
Sep 08 07:05:43.220593 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.220572    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:43.235105 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.235080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vm878" event={"ID":"fc4e3f3b-6bd3-4752-82e3-a373c44729b5","Type":"ContainerStarted","Data":"0204938ce070bf3fb4cdd49f96cc8ffe58df4fbd498cc9dcf1d90a621eb8a19d"}
Sep 08 07:05:43.235513 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.235131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vm878" event={"ID":"fc4e3f3b-6bd3-4752-82e3-a373c44729b5","Type":"ContainerStarted","Data":"13344b9ed0bb00d8329e3beee991b77d729d07ac2467e10e78ef1a4a0276f57b"}
Sep 08 07:05:43.250267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.250242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-tls\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.250370 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.250272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.250370 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.250298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmt2t\" (UniqueName: \"kubernetes.io/projected/040bedf5-aaec-4274-8338-1b97faf7dca3-kube-api-access-wmt2t\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.250450 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.250363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.250450 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.250386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040bedf5-aaec-4274-8338-1b97faf7dca3-metrics-client-ca\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.250450 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.250412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.250450 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.250428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-grpc-tls\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.250587 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.250483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.258718 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.258676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vm878" podStartSLOduration=1.7615602240000001 podStartE2EDuration="3.258661488s" totalImagesPullingTime="1.415249815s" totalInitContainerRuntime="81.851449ms" isStatefulPod=true podCreationTimestamp="2026-09-08 07:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:43.258242669 +0000 UTC m=+58.941702822" watchObservedRunningTime="2026-09-08 07:05:43.258661488 +0000 UTC m=+58.942121640"
Sep 08 07:05:43.299604 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.299571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:43.351197 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.351154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.351375 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.351305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-tls\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.351549 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.351525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.351636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.351597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmt2t\" (UniqueName: \"kubernetes.io/projected/040bedf5-aaec-4274-8338-1b97faf7dca3-kube-api-access-wmt2t\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.351693 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.351679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.351750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.351706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040bedf5-aaec-4274-8338-1b97faf7dca3-metrics-client-ca\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.351807 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.351759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.351807 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.351792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-grpc-tls\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.352405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.352384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040bedf5-aaec-4274-8338-1b97faf7dca3-metrics-client-ca\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.354310 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.354281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-tls\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.354391 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.354341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.354391 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.354341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.354500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.354433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.354648 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.354631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.354685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.354647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/040bedf5-aaec-4274-8338-1b97faf7dca3-secret-grpc-tls\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.358965 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.358944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmt2t\" (UniqueName: \"kubernetes.io/projected/040bedf5-aaec-4274-8338-1b97faf7dca3-kube-api-access-wmt2t\") pod \"thanos-querier-5f6d664c87-2qd5d\" (UID: \"040bedf5-aaec-4274-8338-1b97faf7dca3\") " pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.390321 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.390223    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 07:05:43.392864 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:43.392839    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72124c50_9bed_4bcf_adf4_41e6f33d5720.slice/crio-40eb26b32717b4d279bb5e05c5db877917db9a26e03551a6937bbe2647b69a10 WatchSource:0}: Error finding container 40eb26b32717b4d279bb5e05c5db877917db9a26e03551a6937bbe2647b69a10: Status 404 returned error can't find the container with id 40eb26b32717b4d279bb5e05c5db877917db9a26e03551a6937bbe2647b69a10
Sep 08 07:05:43.513188 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.513101    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:43.587124 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.587097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:43.786153 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:43.786128    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"]
Sep 08 07:05:43.788354 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:43.788324    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040bedf5_aaec_4274_8338_1b97faf7dca3.slice/crio-6a0d7fbad06d9459c9720cbeff48768f4583ff8326231669c2cad24a85041faf WatchSource:0}: Error finding container 6a0d7fbad06d9459c9720cbeff48768f4583ff8326231669c2cad24a85041faf: Status 404 returned error can't find the container with id 6a0d7fbad06d9459c9720cbeff48768f4583ff8326231669c2cad24a85041faf
Sep 08 07:05:44.052306 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.052227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:44.120824 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.120799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpbjh"
Sep 08 07:05:44.151677 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.151623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:44.240105 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.240037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d" event={"ID":"040bedf5-aaec-4274-8338-1b97faf7dca3","Type":"ContainerStarted","Data":"6a0d7fbad06d9459c9720cbeff48768f4583ff8326231669c2cad24a85041faf"}
Sep 08 07:05:44.241529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.241496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerStarted","Data":"40eb26b32717b4d279bb5e05c5db877917db9a26e03551a6937bbe2647b69a10"}
Sep 08 07:05:44.351886 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.351853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:44.409732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.409701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:44.609525 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.609428    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9d98b4687-2grc6"]
Sep 08 07:05:44.612095 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.612058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.614472 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.614432    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 08 07:05:44.615197 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.615175    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 07:05:44.615325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.615214    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 08 07:05:44.615325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.615227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 08 07:05:44.615325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.615283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vtczz\""
Sep 08 07:05:44.615325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.615227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7el54u7iduqgj\""
Sep 08 07:05:44.622999 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.622973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9d98b4687-2grc6"]
Sep 08 07:05:44.664443 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.664410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-secret-metrics-server-client-certs\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.664630 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.664499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lppq\" (UniqueName: \"kubernetes.io/projected/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-kube-api-access-2lppq\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.664630 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.664528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.664630 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.664570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-client-ca-bundle\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.664630 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.664592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-audit-log\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.664852 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.664657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-secret-metrics-server-tls\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.664852 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.664674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-metrics-server-audit-profiles\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.765557 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.765523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-secret-metrics-server-tls\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.765754 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.765566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-metrics-server-audit-profiles\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.765754 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.765633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-secret-metrics-server-client-certs\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.765754 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.765687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lppq\" (UniqueName: \"kubernetes.io/projected/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-kube-api-access-2lppq\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.765754 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.765718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.765754 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.765747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-client-ca-bundle\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.765993 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.765771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-audit-log\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.766287 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.766263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-audit-log\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.768161 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.768135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 08 07:05:44.768263 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.768226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 08 07:05:44.768333 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.768272    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 07:05:44.768556 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.768536    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7el54u7iduqgj\""
Sep 08 07:05:44.768625 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.768606    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 08 07:05:44.775515 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.774285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lppq\" (UniqueName: \"kubernetes.io/projected/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-kube-api-access-2lppq\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.777120 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.777079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.777668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.777646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-metrics-server-audit-profiles\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.779308 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.779160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-secret-metrics-server-client-certs\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.779395 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.779317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-client-ca-bundle\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.779934 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.779911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7ef724af-a951-4c35-aba2-08cf0d9b0b0f-secret-metrics-server-tls\") pod \"metrics-server-9d98b4687-2grc6\" (UID: \"7ef724af-a951-4c35-aba2-08cf0d9b0b0f\") " pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.924917 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.924834    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vtczz\""
Sep 08 07:05:44.933498 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.933454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:05:44.982965 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:44.982898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:45.048993 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.048955    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:45.176632 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.176560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:45.222118 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.222083    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9d98b4687-2grc6"]
Sep 08 07:05:45.226207 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:45.226179    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef724af_a951_4c35_aba2_08cf0d9b0b0f.slice/crio-250f739f1d1c47393b47b0c2b4f73e142c361a14300724971ee0ee11ebd68983 WatchSource:0}: Error finding container 250f739f1d1c47393b47b0c2b4f73e142c361a14300724971ee0ee11ebd68983: Status 404 returned error can't find the container with id 250f739f1d1c47393b47b0c2b4f73e142c361a14300724971ee0ee11ebd68983
Sep 08 07:05:45.246190 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.246164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6" event={"ID":"7ef724af-a951-4c35-aba2-08cf0d9b0b0f","Type":"ContainerStarted","Data":"250f739f1d1c47393b47b0c2b4f73e142c361a14300724971ee0ee11ebd68983"}
Sep 08 07:05:45.247855 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.247831    2580 generic.go:356] "Generic (PLEG): container finished" podID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerID="4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab" exitCode=0
Sep 08 07:05:45.247941 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.247887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerDied","Data":"4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab"}
Sep 08 07:05:45.690186 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.690146    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64f959d977-kwf6h"]
Sep 08 07:05:45.714861 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.714830    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85b5fb64bf-l5qgd"]
Sep 08 07:05:45.718073 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.718053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.724400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.724375    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 07:05:45.729127 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.729102    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b5fb64bf-l5qgd"]
Sep 08 07:05:45.774696 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.774675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-serving-cert\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.774777 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.774705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-config\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.774860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.774797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-oauth-config\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.774860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.774833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-service-ca\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.774860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.774849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrl9w\" (UniqueName: \"kubernetes.io/projected/6f9b26b9-e055-4604-9419-eced7bfe2ebc-kube-api-access-xrl9w\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.774952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.774879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-trusted-ca-bundle\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.774952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.774902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-oauth-serving-cert\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.820153 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.820131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:45.875817 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.875795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-service-ca\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.875909 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.875824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrl9w\" (UniqueName: \"kubernetes.io/projected/6f9b26b9-e055-4604-9419-eced7bfe2ebc-kube-api-access-xrl9w\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.875909 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.875855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-trusted-ca-bundle\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.875909 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.875881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-oauth-serving-cert\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.876004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.875921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-serving-cert\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.876004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.875938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-config\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.876229 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.876205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-oauth-config\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.876707 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.876675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-service-ca\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.876804 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.876738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-config\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.877034 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.876907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-oauth-serving-cert\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.878640 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.877665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-trusted-ca-bundle\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.879235 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.879214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-oauth-config\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.879704 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.879683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-serving-cert\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.888979 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.888955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrl9w\" (UniqueName: \"kubernetes.io/projected/6f9b26b9-e055-4604-9419-eced7bfe2ebc-kube-api-access-xrl9w\") pod \"console-85b5fb64bf-l5qgd\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") " pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:45.921232 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:45.921206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:46.030746 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.030708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:46.203917 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.203790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:46.257259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.257033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d" event={"ID":"040bedf5-aaec-4274-8338-1b97faf7dca3","Type":"ContainerStarted","Data":"bcff0bcc6730f0ac311bf58a945086054bf265a9a669aee18c9e641f34bcc961"}
Sep 08 07:05:46.257259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.257155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d" event={"ID":"040bedf5-aaec-4274-8338-1b97faf7dca3","Type":"ContainerStarted","Data":"041490e1a375d1312e08bce3bfb512ed0522bff2bdbf58d047c7481a7a460123"}
Sep 08 07:05:46.257259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.257200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d" event={"ID":"040bedf5-aaec-4274-8338-1b97faf7dca3","Type":"ContainerStarted","Data":"fbd8befef6eb151c11a5d5971352ef8b4b270f4d20a83cb962821d653bb07dd7"}
Sep 08 07:05:46.302615 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.302586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:46.319378 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.316396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b5fb64bf-l5qgd"]
Sep 08 07:05:46.321667 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:46.321637    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9b26b9_e055_4604_9419_eced7bfe2ebc.slice/crio-5cf8fea1aff7d7d8bbdf33446273a9250b22427847bfb8c3c4d0e7e901ebe471 WatchSource:0}: Error finding container 5cf8fea1aff7d7d8bbdf33446273a9250b22427847bfb8c3c4d0e7e901ebe471: Status 404 returned error can't find the container with id 5cf8fea1aff7d7d8bbdf33446273a9250b22427847bfb8c3c4d0e7e901ebe471
Sep 08 07:05:46.467522 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.463505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:05:46.475507 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.472956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:46.475507 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.474417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.480586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.478944    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 07:05:46.480586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.479162    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 07:05:46.480586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.479572    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 07:05:46.480586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.479772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 07:05:46.480586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.479995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 07:05:46.480586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.480255    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 07:05:46.480586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.480474    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8svz7\""
Sep 08 07:05:46.481036 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.480658    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 07:05:46.481036 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.480838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 07:05:46.481134 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.481085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 07:05:46.484483 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.481254    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 07:05:46.484483 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.481425    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 07:05:46.485720 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.485685    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 07:05:46.492021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.489116    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 07:05:46.492021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.491288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:05:46.492021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.491479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-947hqnqhau2fc\""
Sep 08 07:05:46.492021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.491831    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 07:05:46.587499 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587499 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcsk\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-kube-api-access-vjcsk\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-config\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587835 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587835 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587895 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587925 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.587989 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.587958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.588023 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.588004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.588053 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.588033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.588093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.588057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.588128 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.588105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.588160 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.588130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.588195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.588166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.588225 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.588204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689172 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-config\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.689732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.690062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.690062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.690062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.689974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.690062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.690001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcsk\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-kube-api-access-vjcsk\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.690062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.690049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.691736 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.690933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.691736 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.691530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.692022 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.691955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.693407 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.693379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.694048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.694022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.694594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.694565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.695139 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.695114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.695914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.695780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.695914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.695829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-config\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.696357 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.696331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.696689 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.696661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.697944 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.697882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.698497 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.698358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.698497 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.698444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.698624 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.698510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.699423 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.699394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.700044 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.700015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.700382 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.700356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.700812 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.700793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.702221 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.702200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcsk\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-kube-api-access-vjcsk\") pod \"prometheus-k8s-0\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:46.801751 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:46.801660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:47.251692 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.251653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:47.262357 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.262317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b5fb64bf-l5qgd" event={"ID":"6f9b26b9-e055-4604-9419-eced7bfe2ebc","Type":"ContainerStarted","Data":"07e8438d3b4ccd5a6ec99b1b8bd9badac0df7e404c485f704b1749f9dea7db80"}
Sep 08 07:05:47.262794 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.262385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b5fb64bf-l5qgd" event={"ID":"6f9b26b9-e055-4604-9419-eced7bfe2ebc","Type":"ContainerStarted","Data":"5cf8fea1aff7d7d8bbdf33446273a9250b22427847bfb8c3c4d0e7e901ebe471"}
Sep 08 07:05:47.279478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.279407    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85b5fb64bf-l5qgd" podStartSLOduration=2.279389306 podStartE2EDuration="2.279389306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:47.277788839 +0000 UTC m=+62.961248992" watchObservedRunningTime="2026-09-08 07:05:47.279389306 +0000 UTC m=+62.962849459"
Sep 08 07:05:47.352416 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.352379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:47.407176 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.407145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:47.694124 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.694083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:47.794493 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.794473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:47.870021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:47.870000    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:05:47.873994 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:47.873967    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0405bda_7414_4f10_a66e_82b6bf7c199f.slice/crio-7969c34ec5822905ebc462e2f21ca49a93516ab6aa8fb4670dd2ae68f3f1ae27 WatchSource:0}: Error finding container 7969c34ec5822905ebc462e2f21ca49a93516ab6aa8fb4670dd2ae68f3f1ae27: Status 404 returned error can't find the container with id 7969c34ec5822905ebc462e2f21ca49a93516ab6aa8fb4670dd2ae68f3f1ae27
Sep 08 07:05:48.134296 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.134265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:48.267057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.267012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6" event={"ID":"7ef724af-a951-4c35-aba2-08cf0d9b0b0f","Type":"ContainerStarted","Data":"4db6ae233cbefc27a839def41fbced9ffd98ff7e06834713742f80c45224b4e6"}
Sep 08 07:05:48.269914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.269882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d" event={"ID":"040bedf5-aaec-4274-8338-1b97faf7dca3","Type":"ContainerStarted","Data":"e3623c512d60d0a033be6b62e58b98b2f3fb52b683a376ae89af46d4a59437f7"}
Sep 08 07:05:48.270037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.269954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d" event={"ID":"040bedf5-aaec-4274-8338-1b97faf7dca3","Type":"ContainerStarted","Data":"117925ce0c3a845fec3ca9c58422c4afeae2aab2ae22b1819b5c3a6cc084b43e"}
Sep 08 07:05:48.270037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.270000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d" event={"ID":"040bedf5-aaec-4274-8338-1b97faf7dca3","Type":"ContainerStarted","Data":"ad45b432f70c8733e2ebc99a576d62e3406c75c019b5fdcd859f025ef56c5bf2"}
Sep 08 07:05:48.270037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.270036    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:48.273171 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.273151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerStarted","Data":"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe"}
Sep 08 07:05:48.273256 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.273210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerStarted","Data":"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e"}
Sep 08 07:05:48.273256 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.273249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerStarted","Data":"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8"}
Sep 08 07:05:48.273316 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.273278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerStarted","Data":"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f"}
Sep 08 07:05:48.273316 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.273308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerStarted","Data":"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb"}
Sep 08 07:05:48.273396 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.273331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerStarted","Data":"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200"}
Sep 08 07:05:48.274618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.274596    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a" exitCode=0
Sep 08 07:05:48.274710 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.274622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerDied","Data":"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"}
Sep 08 07:05:48.274710 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.274671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerStarted","Data":"7969c34ec5822905ebc462e2f21ca49a93516ab6aa8fb4670dd2ae68f3f1ae27"}
Sep 08 07:05:48.283872 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.283838    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6" podStartSLOduration=1.9192396729999999 podStartE2EDuration="4.28382599s" totalImagesPullingTime="2.364586317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:48.282878978 +0000 UTC m=+63.966339135" watchObservedRunningTime="2026-09-08 07:05:48.28382599 +0000 UTC m=+63.967286140"
Sep 08 07:05:48.330483 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.330429    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d" podStartSLOduration=1.697873027 podStartE2EDuration="5.330414823s" totalImagesPullingTime="3.632541796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:48.329086991 +0000 UTC m=+64.012547144" watchObservedRunningTime="2026-09-08 07:05:48.330414823 +0000 UTC m=+64.013874977"
Sep 08 07:05:48.352984 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:48.352946    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.981089371 podStartE2EDuration="7.35293329s" totalImagesPullingTime="3.355943684s" totalInitContainerRuntime="15.900235ms" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:48.351080042 +0000 UTC m=+64.034540197" watchObservedRunningTime="2026-09-08 07:05:48.35293329 +0000 UTC m=+64.036393448"
Sep 08 07:05:49.122839 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:49.122809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:49.184435 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:49.184403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:49.623159 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:49.623125    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:49.675324 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:49.675294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:50.533053 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.533014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:50.535369 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.535344    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 07:05:50.546299 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.546273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18397989-832c-4e4d-b309-f09e6c5e0207-metrics-certs\") pod \"network-metrics-daemon-hb6qs\" (UID: \"18397989-832c-4e4d-b309-f09e6c5e0207\") " pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:50.633897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.633863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:50.634309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.633941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:50.636205 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.636165    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 07:05:50.636205 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.636202    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 07:05:50.646568 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.646544    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 07:05:50.647326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.647295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bb1a056e-d8cc-4090-96ef-c2f53cfa99a8-original-pull-secret\") pod \"global-pull-secret-syncer-zrqsh\" (UID: \"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8\") " pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:50.650669 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.650646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ldq7w\""
Sep 08 07:05:50.657703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.657679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtjs\" (UniqueName: \"kubernetes.io/projected/34662fee-4240-4d04-8694-a55dc1510dc4-kube-api-access-hmtjs\") pod \"network-check-target-gtz5m\" (UID: \"34662fee-4240-4d04-8694-a55dc1510dc4\") " pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:50.659246 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.659228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6qs"
Sep 08 07:05:50.661409 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.661391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrqsh"
Sep 08 07:05:50.945600 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.945487    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zrqsh"]
Sep 08 07:05:50.948673 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.948648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hb6qs"]
Sep 08 07:05:50.958110 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.958082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7mcm7\""
Sep 08 07:05:50.966603 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:50.966581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:51.116864 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:51.116829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:51.216719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:51.216629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:51.965120 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:51.965085    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1a056e_d8cc_4090_96ef_c2f53cfa99a8.slice/crio-a496777d3558406d5e6039e19ac1063be5c3d4477918fa30c0b402ad94a0a375 WatchSource:0}: Error finding container a496777d3558406d5e6039e19ac1063be5c3d4477918fa30c0b402ad94a0a375: Status 404 returned error can't find the container with id a496777d3558406d5e6039e19ac1063be5c3d4477918fa30c0b402ad94a0a375
Sep 08 07:05:51.966006 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:51.965949    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18397989_832c_4e4d_b309_f09e6c5e0207.slice/crio-59c5046c833e089f5e4f6dd656966ead23cc942410d9d47c366b967053d8febb WatchSource:0}: Error finding container 59c5046c833e089f5e4f6dd656966ead23cc942410d9d47c366b967053d8febb: Status 404 returned error can't find the container with id 59c5046c833e089f5e4f6dd656966ead23cc942410d9d47c366b967053d8febb
Sep 08 07:05:52.244034 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:52.243818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gtz5m"]
Sep 08 07:05:52.250555 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:05:52.250506    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34662fee_4240_4d04_8694_a55dc1510dc4.slice/crio-855c2dbdcdeac7dc5d7a5724f82953029c9b96fcb17374d5d8f62b0d0bc3e74e WatchSource:0}: Error finding container 855c2dbdcdeac7dc5d7a5724f82953029c9b96fcb17374d5d8f62b0d0bc3e74e: Status 404 returned error can't find the container with id 855c2dbdcdeac7dc5d7a5724f82953029c9b96fcb17374d5d8f62b0d0bc3e74e
Sep 08 07:05:52.297823 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:52.297798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerStarted","Data":"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"}
Sep 08 07:05:52.297952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:52.297875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerStarted","Data":"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"}
Sep 08 07:05:52.297952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:52.297916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerStarted","Data":"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"}
Sep 08 07:05:52.299213 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:52.299188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zrqsh" event={"ID":"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8","Type":"ContainerStarted","Data":"a496777d3558406d5e6039e19ac1063be5c3d4477918fa30c0b402ad94a0a375"}
Sep 08 07:05:52.300529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:52.300501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gtz5m" event={"ID":"34662fee-4240-4d04-8694-a55dc1510dc4","Type":"ContainerStarted","Data":"855c2dbdcdeac7dc5d7a5724f82953029c9b96fcb17374d5d8f62b0d0bc3e74e"}
Sep 08 07:05:52.302541 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:52.302511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hb6qs" event={"ID":"18397989-832c-4e4d-b309-f09e6c5e0207","Type":"ContainerStarted","Data":"59c5046c833e089f5e4f6dd656966ead23cc942410d9d47c366b967053d8febb"}
Sep 08 07:05:53.048840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:53.048762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:53.150243 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:53.150199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:53.312806 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:53.312738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerStarted","Data":"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"}
Sep 08 07:05:53.312937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:53.312821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerStarted","Data":"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"}
Sep 08 07:05:53.312937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:53.312862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerStarted","Data":"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"}
Sep 08 07:05:53.315066 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:53.314954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hb6qs" event={"ID":"18397989-832c-4e4d-b309-f09e6c5e0207","Type":"ContainerStarted","Data":"b68d7a28354ca0d5430dab359d291a57438d52e31aacad5e10fa392c3127fa32"}
Sep 08 07:05:53.363813 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:53.363754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:53.468664 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:53.468628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:54.287367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:54.287335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f6d664c87-2qd5d"
Sep 08 07:05:54.315957 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:54.315874    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.544398053 podStartE2EDuration="8.315854209s" totalImagesPullingTime="3.749802907s" totalInitContainerRuntime="21.653249ms" isStatefulPod=false podCreationTimestamp="2026-09-08 07:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:53.343360528 +0000 UTC m=+69.026820691" watchObservedRunningTime="2026-09-08 07:05:54.315854209 +0000 UTC m=+69.999314369"
Sep 08 07:05:54.335417 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:54.335373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hb6qs" event={"ID":"18397989-832c-4e4d-b309-f09e6c5e0207","Type":"ContainerStarted","Data":"d67eadfc0b7f209d18ee8048af484a99a799168ed5552814de478f564d8d9bde"}
Sep 08 07:05:54.352212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:54.352091    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hb6qs" podStartSLOduration=69.274354946 podStartE2EDuration="1m10.352063795s" totalImagesPullingTime="1.077708849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:54.349956159 +0000 UTC m=+70.033416313" watchObservedRunningTime="2026-09-08 07:05:54.352063795 +0000 UTC m=+70.035523950"
Sep 08 07:05:54.437607 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:54.437569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:54.539253 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:54.539161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:56.031283 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.031249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:56.031720 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.031318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:56.036738 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.036713    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:56.347643 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.347616    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:05:56.492930 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.492897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:56.546732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.546690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:56.601395 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.601319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:56.764793 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.764759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:56.801745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.801716    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:05:56.864490 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:56.864444    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:05:57.348365 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:57.348328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zrqsh" event={"ID":"bb1a056e-d8cc-4090-96ef-c2f53cfa99a8","Type":"ContainerStarted","Data":"8426c7e5008a9d989b8c998add941cf9f8c1f4cb497ad48a79113c1a2d00ef6c"}
Sep 08 07:05:57.350075 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:57.350048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gtz5m" event={"ID":"34662fee-4240-4d04-8694-a55dc1510dc4","Type":"ContainerStarted","Data":"8bb61dbc4833eac39fc6768d1e32669ee304045f06b9ced42c5881fec5aa141d"}
Sep 08 07:05:57.350233 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:57.350216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:05:57.370192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:57.370138    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zrqsh" podStartSLOduration=67.507370384 podStartE2EDuration="1m12.370122998s" totalImagesPullingTime="4.862752614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:57.368202704 +0000 UTC m=+73.051662852" watchObservedRunningTime="2026-09-08 07:05:57.370122998 +0000 UTC m=+73.053583151"
Sep 08 07:05:57.384398 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:05:57.384358    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gtz5m" podStartSLOduration=68.815716264 podStartE2EDuration="1m13.384342391s" totalImagesPullingTime="4.568626127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:05:57.383257177 +0000 UTC m=+73.066717322" watchObservedRunningTime="2026-09-08 07:05:57.384342391 +0000 UTC m=+73.067802544"
Sep 08 07:06:04.625921 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:04.625886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:06:04.729938 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:04.729899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:06:04.938697 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:04.938626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:06:04.938697 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:04.938667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:06:10.715410 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:10.715256    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64f959d977-kwf6h" podUID="66dc7fb9-c7a9-42a1-98ac-6a08557c6514" containerName="console" containerID="cri-o://c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94" gracePeriod=15
Sep 08 07:06:10.957315 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:10.957290    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:06:11.115273 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-oauth-serving-cert\") pod \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") "
Sep 08 07:06:11.115473 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-config\") pod \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") "
Sep 08 07:06:11.115473 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-service-ca\") pod \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") "
Sep 08 07:06:11.115473 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-oauth-config\") pod \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") "
Sep 08 07:06:11.115473 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115388    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-serving-cert\") pod \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") "
Sep 08 07:06:11.115473 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gztbs\" (UniqueName: \"kubernetes.io/projected/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-kube-api-access-gztbs\") pod \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\" (UID: \"66dc7fb9-c7a9-42a1-98ac-6a08557c6514\") "
Sep 08 07:06:11.115785 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66dc7fb9-c7a9-42a1-98ac-6a08557c6514" (UID: "66dc7fb9-c7a9-42a1-98ac-6a08557c6514"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:06:11.115785 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-service-ca" (OuterVolumeSpecName: "service-ca") pod "66dc7fb9-c7a9-42a1-98ac-6a08557c6514" (UID: "66dc7fb9-c7a9-42a1-98ac-6a08557c6514"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:06:11.115911 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.115793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-config" (OuterVolumeSpecName: "console-config") pod "66dc7fb9-c7a9-42a1-98ac-6a08557c6514" (UID: "66dc7fb9-c7a9-42a1-98ac-6a08557c6514"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:06:11.117826 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.117796    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-kube-api-access-gztbs" (OuterVolumeSpecName: "kube-api-access-gztbs") pod "66dc7fb9-c7a9-42a1-98ac-6a08557c6514" (UID: "66dc7fb9-c7a9-42a1-98ac-6a08557c6514"). InnerVolumeSpecName "kube-api-access-gztbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:06:11.118089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.118059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66dc7fb9-c7a9-42a1-98ac-6a08557c6514" (UID: "66dc7fb9-c7a9-42a1-98ac-6a08557c6514"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:06:11.118184 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.118150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66dc7fb9-c7a9-42a1-98ac-6a08557c6514" (UID: "66dc7fb9-c7a9-42a1-98ac-6a08557c6514"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:06:11.216543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.216505    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-oauth-serving-cert\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:06:11.216543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.216536    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-config\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:06:11.216543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.216548    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:06:11.216775 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.216556    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-oauth-config\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:06:11.216775 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.216565    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-console-serving-cert\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:06:11.216775 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.216574    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gztbs\" (UniqueName: \"kubernetes.io/projected/66dc7fb9-c7a9-42a1-98ac-6a08557c6514-kube-api-access-gztbs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:06:11.405111 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.405027    2580 generic.go:356] "Generic (PLEG): container finished" podID="66dc7fb9-c7a9-42a1-98ac-6a08557c6514" containerID="c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94" exitCode=0
Sep 08 07:06:11.405111 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.405103    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f959d977-kwf6h"
Sep 08 07:06:11.405288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.405116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f959d977-kwf6h" event={"ID":"66dc7fb9-c7a9-42a1-98ac-6a08557c6514","Type":"ContainerDied","Data":"c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94"}
Sep 08 07:06:11.405288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.405166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f959d977-kwf6h" event={"ID":"66dc7fb9-c7a9-42a1-98ac-6a08557c6514","Type":"ContainerDied","Data":"ae9d34bfebf2747f4d6c61248626a34e9a68303d81a4571604cea125b2ed4160"}
Sep 08 07:06:11.405288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.405191    2580 scope.go:117] "RemoveContainer" containerID="c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94"
Sep 08 07:06:11.414564 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.414545    2580 scope.go:117] "RemoveContainer" containerID="c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94"
Sep 08 07:06:11.414891 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:06:11.414869    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94\": container with ID starting with c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94 not found: ID does not exist" containerID="c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94"
Sep 08 07:06:11.414943 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.414898    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94"} err="failed to get container status \"c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94\": rpc error: code = NotFound desc = could not find container \"c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94\": container with ID starting with c9f05f7fd45da417ea4baf4017fd405b284b9e1b37731c87f40b2ed11e7b1d94 not found: ID does not exist"
Sep 08 07:06:11.425045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.425019    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64f959d977-kwf6h"]
Sep 08 07:06:11.428605 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:11.428585    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64f959d977-kwf6h"]
Sep 08 07:06:12.940234 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:12.940197    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66dc7fb9-c7a9-42a1-98ac-6a08557c6514" path="/var/lib/kubelet/pods/66dc7fb9-c7a9-42a1-98ac-6a08557c6514/volumes"
Sep 08 07:06:19.289945 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:19.289873    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:06:19.347688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:19.347658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:06:19.627888 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:19.627858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:06:19.810448 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:19.810415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:06:24.943813 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:24.943782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:06:24.947539 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:24.947515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9d98b4687-2grc6"
Sep 08 07:06:25.536642 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:25.536607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:06:28.356996 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:28.356967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gtz5m"
Sep 08 07:06:46.802719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:46.802680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:06:46.822148 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:46.822124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:06:47.560781 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:47.560748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:06:59.447325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:59.447277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:06:59.561981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:06:59.561947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:07:00.496230 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:00.496189    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 07:07:00.496753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:00.496593    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="alertmanager" containerID="cri-o://7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200" gracePeriod=120
Sep 08 07:07:00.496753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:00.496648    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy-metric" containerID="cri-o://4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e" gracePeriod=120
Sep 08 07:07:00.496753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:00.496658    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy-web" containerID="cri-o://9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f" gracePeriod=120
Sep 08 07:07:00.496753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:00.496679    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy" containerID="cri-o://71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8" gracePeriod=120
Sep 08 07:07:00.496753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:00.496710    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="prom-label-proxy" containerID="cri-o://9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe" gracePeriod=120
Sep 08 07:07:00.496976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:00.496717    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="config-reloader" containerID="cri-o://2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb" gracePeriod=120
Sep 08 07:07:01.606137 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.606110    2580 generic.go:356] "Generic (PLEG): container finished" podID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerID="9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe" exitCode=0
Sep 08 07:07:01.606137 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.606134    2580 generic.go:356] "Generic (PLEG): container finished" podID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerID="71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8" exitCode=0
Sep 08 07:07:01.606137 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.606142    2580 generic.go:356] "Generic (PLEG): container finished" podID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerID="2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb" exitCode=0
Sep 08 07:07:01.606614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.606149    2580 generic.go:356] "Generic (PLEG): container finished" podID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerID="7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200" exitCode=0
Sep 08 07:07:01.606614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.606179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerDied","Data":"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe"}
Sep 08 07:07:01.606614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.606248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerDied","Data":"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8"}
Sep 08 07:07:01.606614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.606273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerDied","Data":"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb"}
Sep 08 07:07:01.606614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.606301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerDied","Data":"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200"}
Sep 08 07:07:01.732036 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.732012    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:07:01.838570 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.838483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-metric\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.838570 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.838525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-main-db\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.838765 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.838576    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.838765 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.838606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-out\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.838765 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.838630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-volume\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.838765 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.838667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-cluster-tls-config\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.838765 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.838713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-metrics-client-ca\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.839028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.838956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:07:01.839089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839045    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.839089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839068    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:01.839089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-tls-assets\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.839238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-web-config\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.839238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4r7b\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-kube-api-access-m4r7b\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.839238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-web\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.839238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839225    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle\") pod \"72124c50-9bed-4bcf-adf4-41e6f33d5720\" (UID: \"72124c50-9bed-4bcf-adf4-41e6f33d5720\") "
Sep 08 07:07:01.839911 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839475    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-main-db\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.839911 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839500    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-metrics-client-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.839911 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.839856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:01.841936 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.841892    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:01.842271 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.842219    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:01.842588 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.842521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:01.842674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.842585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-volume" (OuterVolumeSpecName: "config-volume") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:01.842780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.842699    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-kube-api-access-m4r7b" (OuterVolumeSpecName: "kube-api-access-m4r7b") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "kube-api-access-m4r7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:07:01.842780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.842714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-out" (OuterVolumeSpecName: "config-out") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:07:01.842780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.842754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:01.843109 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.843084    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:07:01.846074 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.845981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:01.851932 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.851910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-web-config" (OuterVolumeSpecName: "web-config") pod "72124c50-9bed-4bcf-adf4-41e6f33d5720" (UID: "72124c50-9bed-4bcf-adf4-41e6f33d5720"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:01.940831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940798    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.940831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940829    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72124c50-9bed-4bcf-adf4-41e6f33d5720-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.940831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940839    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.941048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940848    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.941048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940859    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-out\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.941048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940867    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.941048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940876    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-cluster-tls-config\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.941048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940884    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-secret-alertmanager-main-tls\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.941048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940892    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-tls-assets\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.941048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940900    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72124c50-9bed-4bcf-adf4-41e6f33d5720-web-config\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:01.941048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:01.940907    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4r7b\" (UniqueName: \"kubernetes.io/projected/72124c50-9bed-4bcf-adf4-41e6f33d5720-kube-api-access-m4r7b\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:02.617743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.617706    2580 generic.go:356] "Generic (PLEG): container finished" podID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerID="4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e" exitCode=0
Sep 08 07:07:02.617743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.617734    2580 generic.go:356] "Generic (PLEG): container finished" podID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerID="9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f" exitCode=0
Sep 08 07:07:02.617743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.617738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerDied","Data":"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e"}
Sep 08 07:07:02.618248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.617790    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 07:07:02.618248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.617832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerDied","Data":"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f"}
Sep 08 07:07:02.618248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.617860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"72124c50-9bed-4bcf-adf4-41e6f33d5720","Type":"ContainerDied","Data":"40eb26b32717b4d279bb5e05c5db877917db9a26e03551a6937bbe2647b69a10"}
Sep 08 07:07:02.618248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.617890    2580 scope.go:117] "RemoveContainer" containerID="9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe"
Sep 08 07:07:02.626402 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.626380    2580 scope.go:117] "RemoveContainer" containerID="4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e"
Sep 08 07:07:02.633424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.633405    2580 scope.go:117] "RemoveContainer" containerID="71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8"
Sep 08 07:07:02.641259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.641235    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 07:07:02.641259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.641239    2580 scope.go:117] "RemoveContainer" containerID="9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f"
Sep 08 07:07:02.645880 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.645856    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 07:07:02.649810 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.649786    2580 scope.go:117] "RemoveContainer" containerID="2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb"
Sep 08 07:07:02.656574 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.656555    2580 scope.go:117] "RemoveContainer" containerID="7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200"
Sep 08 07:07:02.663870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.663854    2580 scope.go:117] "RemoveContainer" containerID="4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab"
Sep 08 07:07:02.670746 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.670725    2580 scope.go:117] "RemoveContainer" containerID="9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe"
Sep 08 07:07:02.671028 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:02.671006    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe\": container with ID starting with 9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe not found: ID does not exist" containerID="9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe"
Sep 08 07:07:02.671135 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.671039    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe"} err="failed to get container status \"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe\": rpc error: code = NotFound desc = could not find container \"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe\": container with ID starting with 9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe not found: ID does not exist"
Sep 08 07:07:02.671135 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.671064    2580 scope.go:117] "RemoveContainer" containerID="4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e"
Sep 08 07:07:02.671366 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:02.671347    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e\": container with ID starting with 4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e not found: ID does not exist" containerID="4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e"
Sep 08 07:07:02.671407 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.671371    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e"} err="failed to get container status \"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e\": rpc error: code = NotFound desc = could not find container \"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e\": container with ID starting with 4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e not found: ID does not exist"
Sep 08 07:07:02.671407 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.671392    2580 scope.go:117] "RemoveContainer" containerID="71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8"
Sep 08 07:07:02.671720 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:02.671702    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8\": container with ID starting with 71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8 not found: ID does not exist" containerID="71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8"
Sep 08 07:07:02.671778 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.671725    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8"} err="failed to get container status \"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8\": rpc error: code = NotFound desc = could not find container \"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8\": container with ID starting with 71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8 not found: ID does not exist"
Sep 08 07:07:02.671778 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.671740    2580 scope.go:117] "RemoveContainer" containerID="9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f"
Sep 08 07:07:02.671977 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:02.671960    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f\": container with ID starting with 9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f not found: ID does not exist" containerID="9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f"
Sep 08 07:07:02.672023 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.671982    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f"} err="failed to get container status \"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f\": rpc error: code = NotFound desc = could not find container \"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f\": container with ID starting with 9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f not found: ID does not exist"
Sep 08 07:07:02.672023 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.671994    2580 scope.go:117] "RemoveContainer" containerID="2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb"
Sep 08 07:07:02.672242 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:02.672226    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb\": container with ID starting with 2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb not found: ID does not exist" containerID="2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb"
Sep 08 07:07:02.672283 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.672245    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb"} err="failed to get container status \"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb\": rpc error: code = NotFound desc = could not find container \"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb\": container with ID starting with 2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb not found: ID does not exist"
Sep 08 07:07:02.672283 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.672259    2580 scope.go:117] "RemoveContainer" containerID="7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200"
Sep 08 07:07:02.672486 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:02.672452    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200\": container with ID starting with 7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200 not found: ID does not exist" containerID="7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200"
Sep 08 07:07:02.672542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.672494    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200"} err="failed to get container status \"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200\": rpc error: code = NotFound desc = could not find container \"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200\": container with ID starting with 7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200 not found: ID does not exist"
Sep 08 07:07:02.672542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.672510    2580 scope.go:117] "RemoveContainer" containerID="4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab"
Sep 08 07:07:02.672770 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:02.672747    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab\": container with ID starting with 4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab not found: ID does not exist" containerID="4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab"
Sep 08 07:07:02.672825 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.672774    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab"} err="failed to get container status \"4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab\": rpc error: code = NotFound desc = could not find container \"4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab\": container with ID starting with 4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab not found: ID does not exist"
Sep 08 07:07:02.672825 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.672789    2580 scope.go:117] "RemoveContainer" containerID="9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe"
Sep 08 07:07:02.673028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673011    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe"} err="failed to get container status \"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe\": rpc error: code = NotFound desc = could not find container \"9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe\": container with ID starting with 9aef6fbc0efa1054aacc9e2966547b1ba151361f8ca3c79b64af1ef189f2d6fe not found: ID does not exist"
Sep 08 07:07:02.673076 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673028    2580 scope.go:117] "RemoveContainer" containerID="4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e"
Sep 08 07:07:02.673250 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673230    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e"} err="failed to get container status \"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e\": rpc error: code = NotFound desc = could not find container \"4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e\": container with ID starting with 4da1b45c6f837c1ec65bc7d9fdf3f22626fe02708d399435ce6bfe0012e3226e not found: ID does not exist"
Sep 08 07:07:02.673318 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673253    2580 scope.go:117] "RemoveContainer" containerID="71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8"
Sep 08 07:07:02.673495 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673480    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8"} err="failed to get container status \"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8\": rpc error: code = NotFound desc = could not find container \"71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8\": container with ID starting with 71b014f65b37e5fa82bddc487ddcca072a7ebe3c0c447b496819d120bcb094d8 not found: ID does not exist"
Sep 08 07:07:02.673545 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673496    2580 scope.go:117] "RemoveContainer" containerID="9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f"
Sep 08 07:07:02.673717 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673700    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f"} err="failed to get container status \"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f\": rpc error: code = NotFound desc = could not find container \"9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f\": container with ID starting with 9603a5f5bc7219991b4f3e46764b3bd7c6e4572ca1a07e6e3f08a2732858131f not found: ID does not exist"
Sep 08 07:07:02.673781 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673718    2580 scope.go:117] "RemoveContainer" containerID="2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb"
Sep 08 07:07:02.674014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.673992    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb"} err="failed to get container status \"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb\": rpc error: code = NotFound desc = could not find container \"2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb\": container with ID starting with 2f57341cfca2556c6ce4b7f3a39eb258fe36c6ddda2d3716df4d1de58fe2c6fb not found: ID does not exist"
Sep 08 07:07:02.674014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.674015    2580 scope.go:117] "RemoveContainer" containerID="7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200"
Sep 08 07:07:02.676057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.676030    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200"} err="failed to get container status \"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200\": rpc error: code = NotFound desc = could not find container \"7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200\": container with ID starting with 7ae9f68c56263dbfa60288e8fe7d221679527741281876730c1dddf7ca58d200 not found: ID does not exist"
Sep 08 07:07:02.676057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.676057    2580 scope.go:117] "RemoveContainer" containerID="4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab"
Sep 08 07:07:02.676337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.676316    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab"} err="failed to get container status \"4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab\": rpc error: code = NotFound desc = could not find container \"4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab\": container with ID starting with 4b82e85bd11001de552a1aaf435baa406ee0a6c1e69e993f788951358e966fab not found: ID does not exist"
Sep 08 07:07:02.938896 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:02.938821    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" path="/var/lib/kubelet/pods/72124c50-9bed-4bcf-adf4-41e6f33d5720/volumes"
Sep 08 07:07:04.827297 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:04.827260    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:07:04.827713 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:04.827654    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="prometheus" containerID="cri-o://d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac" gracePeriod=600
Sep 08 07:07:04.827713 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:04.827676    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy" containerID="cri-o://75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be" gracePeriod=600
Sep 08 07:07:04.827713 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:04.827696    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="thanos-sidecar" containerID="cri-o://308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb" gracePeriod=600
Sep 08 07:07:04.827893 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:04.827713    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy-web" containerID="cri-o://e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785" gracePeriod=600
Sep 08 07:07:04.827893 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:04.827725    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="config-reloader" containerID="cri-o://4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2" gracePeriod=600
Sep 08 07:07:04.827893 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:04.827737    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d" gracePeriod=600
Sep 08 07:07:05.063629 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.063603    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.164827 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.164745    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-thanos-prometheus-http-client-file\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.164827 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.164786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-tls\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.164827 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.164815    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-metrics-client-certs\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.164853    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-metrics-client-ca\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.164878    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-tls-assets\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.164905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.164929    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-serving-certs-ca-bundle\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.164998    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-1\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjcsk\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-kube-api-access-vjcsk\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-grpc-tls\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-config\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-web-config\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165213    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-2\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165237    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-kubelet-serving-ca-bundle\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165269    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-trusted-ca-bundle\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-db\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-config-out\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-0\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-kube-rbac-proxy\") pod \"f0405bda-7414-4f10-a66e-82b6bf7c199f\" (UID: \"f0405bda-7414-4f10-a66e-82b6bf7c199f\") "
Sep 08 07:07:05.165999 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165821    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:05.165999 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.165826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:05.166976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.166835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:07:05.167496 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.167436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:05.167797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.167765    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.167896 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.167832    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:05.167974 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.167952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.168477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.168426    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.168590 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.168565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.168829 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.168797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.168829 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.168812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:05.168999 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.168966    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:05.168999 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.168973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:05.169385 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.169361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:07:05.169509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.169390    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-config" (OuterVolumeSpecName: "config") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.169913 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.169874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.170042 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.170022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-config-out" (OuterVolumeSpecName: "config-out") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:07:05.170095 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.170053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.170095 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.170083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-kube-api-access-vjcsk" (OuterVolumeSpecName: "kube-api-access-vjcsk") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "kube-api-access-vjcsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:07:05.178831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.178812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-web-config" (OuterVolumeSpecName: "web-config") pod "f0405bda-7414-4f10-a66e-82b6bf7c199f" (UID: "f0405bda-7414-4f10-a66e-82b6bf7c199f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:05.266336 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266312    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-config-out\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266336 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266335    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266345    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-kube-rbac-proxy\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266355    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266365    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266375    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-metrics-client-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266383    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-metrics-client-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266392    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-tls-assets\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266400    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266409    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266418    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266425    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjcsk\" (UniqueName: \"kubernetes.io/projected/f0405bda-7414-4f10-a66e-82b6bf7c199f-kube-api-access-vjcsk\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266434    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-grpc-tls\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266442    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266451    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-config\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266471    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0405bda-7414-4f10-a66e-82b6bf7c199f-web-config\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266482    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266490    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266498    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.266523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.266506    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0405bda-7414-4f10-a66e-82b6bf7c199f-prometheus-k8s-db\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:05.631723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631691    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d" exitCode=0
Sep 08 07:07:05.631723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631717    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be" exitCode=0
Sep 08 07:07:05.631723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631724    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785" exitCode=0
Sep 08 07:07:05.631723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631730    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb" exitCode=0
Sep 08 07:07:05.631983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631735    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2" exitCode=0
Sep 08 07:07:05.631983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631739    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac" exitCode=0
Sep 08 07:07:05.631983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631781    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.631983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerDied","Data":"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"}
Sep 08 07:07:05.631983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerDied","Data":"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"}
Sep 08 07:07:05.631983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerDied","Data":"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"}
Sep 08 07:07:05.631983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631939    2580 scope.go:117] "RemoveContainer" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"
Sep 08 07:07:05.632228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.631938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerDied","Data":"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"}
Sep 08 07:07:05.632228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.632038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerDied","Data":"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"}
Sep 08 07:07:05.632228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.632087    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerDied","Data":"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"}
Sep 08 07:07:05.632228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.632122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0405bda-7414-4f10-a66e-82b6bf7c199f","Type":"ContainerDied","Data":"7969c34ec5822905ebc462e2f21ca49a93516ab6aa8fb4670dd2ae68f3f1ae27"}
Sep 08 07:07:05.639859 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.639842    2580 scope.go:117] "RemoveContainer" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"
Sep 08 07:07:05.646509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.646491    2580 scope.go:117] "RemoveContainer" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"
Sep 08 07:07:05.652677 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.652662    2580 scope.go:117] "RemoveContainer" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"
Sep 08 07:07:05.656253 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.656233    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:07:05.659244 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.659222    2580 scope.go:117] "RemoveContainer" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"
Sep 08 07:07:05.660845 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.660821    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:07:05.666267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.666250    2580 scope.go:117] "RemoveContainer" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"
Sep 08 07:07:05.672912 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.672896    2580 scope.go:117] "RemoveContainer" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"
Sep 08 07:07:05.679247 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.679233    2580 scope.go:117] "RemoveContainer" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"
Sep 08 07:07:05.679581 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:05.679565    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": container with ID starting with 1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d not found: ID does not exist" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"
Sep 08 07:07:05.679636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.679590    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"} err="failed to get container status \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": rpc error: code = NotFound desc = could not find container \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": container with ID starting with 1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d not found: ID does not exist"
Sep 08 07:07:05.679636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.679606    2580 scope.go:117] "RemoveContainer" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"
Sep 08 07:07:05.679865 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:05.679847    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": container with ID starting with 75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be not found: ID does not exist" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"
Sep 08 07:07:05.679902 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.679872    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"} err="failed to get container status \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": rpc error: code = NotFound desc = could not find container \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": container with ID starting with 75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be not found: ID does not exist"
Sep 08 07:07:05.679902 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.679888    2580 scope.go:117] "RemoveContainer" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"
Sep 08 07:07:05.680142 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:05.680125    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": container with ID starting with e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785 not found: ID does not exist" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"
Sep 08 07:07:05.680195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.680147    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"} err="failed to get container status \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": rpc error: code = NotFound desc = could not find container \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": container with ID starting with e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785 not found: ID does not exist"
Sep 08 07:07:05.680195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.680161    2580 scope.go:117] "RemoveContainer" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"
Sep 08 07:07:05.680402 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:05.680389    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": container with ID starting with 308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb not found: ID does not exist" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"
Sep 08 07:07:05.680441 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.680407    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"} err="failed to get container status \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": rpc error: code = NotFound desc = could not find container \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": container with ID starting with 308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb not found: ID does not exist"
Sep 08 07:07:05.680441 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.680424    2580 scope.go:117] "RemoveContainer" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"
Sep 08 07:07:05.680640 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:05.680625    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": container with ID starting with 4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2 not found: ID does not exist" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"
Sep 08 07:07:05.680674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.680643    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"} err="failed to get container status \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": rpc error: code = NotFound desc = could not find container \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": container with ID starting with 4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2 not found: ID does not exist"
Sep 08 07:07:05.680674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.680653    2580 scope.go:117] "RemoveContainer" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"
Sep 08 07:07:05.680905 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:05.680888    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": container with ID starting with d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac not found: ID does not exist" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"
Sep 08 07:07:05.680953 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.680908    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"} err="failed to get container status \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": rpc error: code = NotFound desc = could not find container \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": container with ID starting with d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac not found: ID does not exist"
Sep 08 07:07:05.680953 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.680921    2580 scope.go:117] "RemoveContainer" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"
Sep 08 07:07:05.681165 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:07:05.681150    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": container with ID starting with ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a not found: ID does not exist" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"
Sep 08 07:07:05.681203 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.681168    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"} err="failed to get container status \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": rpc error: code = NotFound desc = could not find container \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": container with ID starting with ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a not found: ID does not exist"
Sep 08 07:07:05.681203 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.681187    2580 scope.go:117] "RemoveContainer" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"
Sep 08 07:07:05.681409 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.681391    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"} err="failed to get container status \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": rpc error: code = NotFound desc = could not find container \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": container with ID starting with 1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d not found: ID does not exist"
Sep 08 07:07:05.681409 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.681409    2580 scope.go:117] "RemoveContainer" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"
Sep 08 07:07:05.681669 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.681654    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"} err="failed to get container status \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": rpc error: code = NotFound desc = could not find container \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": container with ID starting with 75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be not found: ID does not exist"
Sep 08 07:07:05.681669 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.681668    2580 scope.go:117] "RemoveContainer" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"
Sep 08 07:07:05.681881 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.681854    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"} err="failed to get container status \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": rpc error: code = NotFound desc = could not find container \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": container with ID starting with e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785 not found: ID does not exist"
Sep 08 07:07:05.681923 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.681881    2580 scope.go:117] "RemoveContainer" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"
Sep 08 07:07:05.682095 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682079    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"} err="failed to get container status \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": rpc error: code = NotFound desc = could not find container \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": container with ID starting with 308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb not found: ID does not exist"
Sep 08 07:07:05.682138 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682095    2580 scope.go:117] "RemoveContainer" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"
Sep 08 07:07:05.682280 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682265    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"} err="failed to get container status \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": rpc error: code = NotFound desc = could not find container \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": container with ID starting with 4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2 not found: ID does not exist"
Sep 08 07:07:05.682320 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682280    2580 scope.go:117] "RemoveContainer" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"
Sep 08 07:07:05.682524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682509    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"} err="failed to get container status \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": rpc error: code = NotFound desc = could not find container \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": container with ID starting with d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac not found: ID does not exist"
Sep 08 07:07:05.682524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682523    2580 scope.go:117] "RemoveContainer" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"
Sep 08 07:07:05.682719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682703    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"} err="failed to get container status \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": rpc error: code = NotFound desc = could not find container \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": container with ID starting with ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a not found: ID does not exist"
Sep 08 07:07:05.682719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682718    2580 scope.go:117] "RemoveContainer" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"
Sep 08 07:07:05.682926 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682910    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"} err="failed to get container status \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": rpc error: code = NotFound desc = could not find container \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": container with ID starting with 1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d not found: ID does not exist"
Sep 08 07:07:05.682926 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.682925    2580 scope.go:117] "RemoveContainer" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"
Sep 08 07:07:05.683128 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683113    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"} err="failed to get container status \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": rpc error: code = NotFound desc = could not find container \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": container with ID starting with 75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be not found: ID does not exist"
Sep 08 07:07:05.683169 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683127    2580 scope.go:117] "RemoveContainer" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"
Sep 08 07:07:05.683304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683289    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"} err="failed to get container status \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": rpc error: code = NotFound desc = could not find container \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": container with ID starting with e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785 not found: ID does not exist"
Sep 08 07:07:05.683341 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683304    2580 scope.go:117] "RemoveContainer" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"
Sep 08 07:07:05.683522 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683505    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"} err="failed to get container status \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": rpc error: code = NotFound desc = could not find container \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": container with ID starting with 308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb not found: ID does not exist"
Sep 08 07:07:05.683559 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683522    2580 scope.go:117] "RemoveContainer" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"
Sep 08 07:07:05.683754 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683734    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"} err="failed to get container status \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": rpc error: code = NotFound desc = could not find container \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": container with ID starting with 4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2 not found: ID does not exist"
Sep 08 07:07:05.683805 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683755    2580 scope.go:117] "RemoveContainer" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"
Sep 08 07:07:05.683987 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683970    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"} err="failed to get container status \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": rpc error: code = NotFound desc = could not find container \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": container with ID starting with d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac not found: ID does not exist"
Sep 08 07:07:05.684024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.683988    2580 scope.go:117] "RemoveContainer" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"
Sep 08 07:07:05.684186 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.684162    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"} err="failed to get container status \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": rpc error: code = NotFound desc = could not find container \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": container with ID starting with ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a not found: ID does not exist"
Sep 08 07:07:05.684230 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.684186    2580 scope.go:117] "RemoveContainer" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"
Sep 08 07:07:05.684391 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.684373    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"} err="failed to get container status \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": rpc error: code = NotFound desc = could not find container \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": container with ID starting with 1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d not found: ID does not exist"
Sep 08 07:07:05.684438 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.684391    2580 scope.go:117] "RemoveContainer" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"
Sep 08 07:07:05.684635 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.684610    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"} err="failed to get container status \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": rpc error: code = NotFound desc = could not find container \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": container with ID starting with 75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be not found: ID does not exist"
Sep 08 07:07:05.684675 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.684635    2580 scope.go:117] "RemoveContainer" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"
Sep 08 07:07:05.684822 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.684806    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"} err="failed to get container status \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": rpc error: code = NotFound desc = could not find container \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": container with ID starting with e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785 not found: ID does not exist"
Sep 08 07:07:05.684862 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.684824    2580 scope.go:117] "RemoveContainer" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"
Sep 08 07:07:05.685031 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685017    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"} err="failed to get container status \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": rpc error: code = NotFound desc = could not find container \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": container with ID starting with 308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb not found: ID does not exist"
Sep 08 07:07:05.685031 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685030    2580 scope.go:117] "RemoveContainer" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"
Sep 08 07:07:05.685196 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685174    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"} err="failed to get container status \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": rpc error: code = NotFound desc = could not find container \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": container with ID starting with 4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2 not found: ID does not exist"
Sep 08 07:07:05.685231 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685195    2580 scope.go:117] "RemoveContainer" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"
Sep 08 07:07:05.685386 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685370    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"} err="failed to get container status \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": rpc error: code = NotFound desc = could not find container \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": container with ID starting with d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac not found: ID does not exist"
Sep 08 07:07:05.685424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685386    2580 scope.go:117] "RemoveContainer" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"
Sep 08 07:07:05.685653 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685630    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"} err="failed to get container status \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": rpc error: code = NotFound desc = could not find container \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": container with ID starting with ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a not found: ID does not exist"
Sep 08 07:07:05.685653 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685652    2580 scope.go:117] "RemoveContainer" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"
Sep 08 07:07:05.685958 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685934    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"} err="failed to get container status \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": rpc error: code = NotFound desc = could not find container \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": container with ID starting with 1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d not found: ID does not exist"
Sep 08 07:07:05.686063 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.685958    2580 scope.go:117] "RemoveContainer" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"
Sep 08 07:07:05.686436 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.686407    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"} err="failed to get container status \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": rpc error: code = NotFound desc = could not find container \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": container with ID starting with 75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be not found: ID does not exist"
Sep 08 07:07:05.686543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.686438    2580 scope.go:117] "RemoveContainer" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"
Sep 08 07:07:05.686777 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.686753    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"} err="failed to get container status \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": rpc error: code = NotFound desc = could not find container \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": container with ID starting with e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785 not found: ID does not exist"
Sep 08 07:07:05.686827 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.686780    2580 scope.go:117] "RemoveContainer" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"
Sep 08 07:07:05.687037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687019    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"} err="failed to get container status \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": rpc error: code = NotFound desc = could not find container \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": container with ID starting with 308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb not found: ID does not exist"
Sep 08 07:07:05.687111 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687038    2580 scope.go:117] "RemoveContainer" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"
Sep 08 07:07:05.687249 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687231    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"} err="failed to get container status \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": rpc error: code = NotFound desc = could not find container \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": container with ID starting with 4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2 not found: ID does not exist"
Sep 08 07:07:05.687309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687250    2580 scope.go:117] "RemoveContainer" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"
Sep 08 07:07:05.687561 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687535    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:07:05.687561 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"} err="failed to get container status \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": rpc error: code = NotFound desc = could not find container \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": container with ID starting with d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac not found: ID does not exist"
Sep 08 07:07:05.687678 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687572    2580 scope.go:117] "RemoveContainer" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"
Sep 08 07:07:05.687784 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687772    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="prometheus"
Sep 08 07:07:05.687830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687785    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="prometheus"
Sep 08 07:07:05.687830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687795    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy"
Sep 08 07:07:05.687830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687803    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy"
Sep 08 07:07:05.687830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687817    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66dc7fb9-c7a9-42a1-98ac-6a08557c6514" containerName="console"
Sep 08 07:07:05.687830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687823    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66dc7fb9-c7a9-42a1-98ac-6a08557c6514" containerName="console"
Sep 08 07:07:05.687830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687830    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="config-reloader"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687835    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="config-reloader"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687830    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"} err="failed to get container status \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": rpc error: code = NotFound desc = could not find container \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": container with ID starting with ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a not found: ID does not exist"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687846    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="alertmanager"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687848    2580 scope.go:117] "RemoveContainer" containerID="1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687851    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="alertmanager"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687905    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="config-reloader"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687912    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="config-reloader"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687922    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy-web"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687927    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy-web"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687934    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="init-config-reloader"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687939    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="init-config-reloader"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687948    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy-web"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687953    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy-web"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687960    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687966    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687975    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="init-config-reloader"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687982    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="init-config-reloader"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687989    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="thanos-sidecar"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.687996    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="thanos-sidecar"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688006    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy-metric"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688013    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy-metric"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688023    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="prom-label-proxy"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688031    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="prom-label-proxy"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688039    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy-thanos"
Sep 08 07:07:05.688057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688047    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy-thanos"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688108    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688117    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy-web"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688122    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy-metric"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688129    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="config-reloader"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688135    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="config-reloader"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688141    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="thanos-sidecar"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688145    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy-web"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688150    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="kube-rbac-proxy"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688155    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="alertmanager"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688160    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="prometheus"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688166    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" containerName="kube-rbac-proxy-thanos"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688171    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72124c50-9bed-4bcf-adf4-41e6f33d5720" containerName="prom-label-proxy"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688185    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66dc7fb9-c7a9-42a1-98ac-6a08557c6514" containerName="console"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688155    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d"} err="failed to get container status \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": rpc error: code = NotFound desc = could not find container \"1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d\": container with ID starting with 1c5536c1ba359d63406fd8bb172b5c927fd4e306b9b98b3397163e852dd6b39d not found: ID does not exist"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688238    2580 scope.go:117] "RemoveContainer" containerID="75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688492    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be"} err="failed to get container status \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": rpc error: code = NotFound desc = could not find container \"75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be\": container with ID starting with 75ecc9cfcd8c08f40044850983190146f266c264c3eb16f5a05c1dc534f581be not found: ID does not exist"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688509    2580 scope.go:117] "RemoveContainer" containerID="e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688730    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785"} err="failed to get container status \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": rpc error: code = NotFound desc = could not find container \"e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785\": container with ID starting with e04535adae4a34979a6b67825bc936005bf0e156fa89b126bbb8c87d2ffab785 not found: ID does not exist"
Sep 08 07:07:05.688981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688745    2580 scope.go:117] "RemoveContainer" containerID="308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"
Sep 08 07:07:05.689733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.688990    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb"} err="failed to get container status \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": rpc error: code = NotFound desc = could not find container \"308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb\": container with ID starting with 308d5eb1b794f10f2ea1730bcd0a6dbe831a89d7b76a46263c7478d593debddb not found: ID does not exist"
Sep 08 07:07:05.689733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.689010    2580 scope.go:117] "RemoveContainer" containerID="4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"
Sep 08 07:07:05.689733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.689242    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2"} err="failed to get container status \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": rpc error: code = NotFound desc = could not find container \"4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2\": container with ID starting with 4552641fc77a1d555e35c70ecdcbcb312480491feed4776a9009fa201bab6ff2 not found: ID does not exist"
Sep 08 07:07:05.689733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.689256    2580 scope.go:117] "RemoveContainer" containerID="d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"
Sep 08 07:07:05.689733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.689516    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac"} err="failed to get container status \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": rpc error: code = NotFound desc = could not find container \"d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac\": container with ID starting with d4a83ecbd902af0171c7c3794e82f4bdc5fb930124193721980790e4b7ab6fac not found: ID does not exist"
Sep 08 07:07:05.689733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.689533    2580 scope.go:117] "RemoveContainer" containerID="ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"
Sep 08 07:07:05.689911 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.689736    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a"} err="failed to get container status \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": rpc error: code = NotFound desc = could not find container \"ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a\": container with ID starting with ba0605331df86b47591d1bb9edee383997231f2e34fc065a8e993f7410e53f5a not found: ID does not exist"
Sep 08 07:07:05.691907 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.691892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.694268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694250    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 07:07:05.694347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 07:07:05.694595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694567    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8svz7\""
Sep 08 07:07:05.694595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 07:07:05.694766 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 07:07:05.694876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 07:07:05.694940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 07:07:05.694940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 07:07:05.695103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694880    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 07:07:05.695103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.694995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 07:07:05.695238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.695144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 07:07:05.695352 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.695314    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-947hqnqhau2fc\""
Sep 08 07:07:05.695425 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.695371    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 07:07:05.695525 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.695423    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 07:07:05.697287 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.697266    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 07:07:05.699592 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.699575    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 07:07:05.705868 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.705850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:07:05.872390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2583030-8f67-49b4-8b2b-5f2e08a006ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2583030-8f67-49b4-8b2b-5f2e08a006ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.872797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.873157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.873157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.873157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.873157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.873157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.873157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.872985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.873157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.873008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-config\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.873157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.873050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzk6\" (UniqueName: \"kubernetes.io/projected/d2583030-8f67-49b4-8b2b-5f2e08a006ed-kube-api-access-pgzk6\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.973940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.973848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.973940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.973883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.973940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.973910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-config\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzk6\" (UniqueName: \"kubernetes.io/projected/d2583030-8f67-49b4-8b2b-5f2e08a006ed-kube-api-access-pgzk6\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2583030-8f67-49b4-8b2b-5f2e08a006ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2583030-8f67-49b4-8b2b-5f2e08a006ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.974764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.974736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.975566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.975418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.976070 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.975737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.976070 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.975863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.976249 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.976225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.976688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.976664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.976846 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.976821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.977093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.977070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.977176 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.977131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.977553 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.977507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.978339 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.978275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.978848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.978806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2583030-8f67-49b4-8b2b-5f2e08a006ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.978976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.978955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2583030-8f67-49b4-8b2b-5f2e08a006ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.979524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.979358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-config\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.979524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.979399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2583030-8f67-49b4-8b2b-5f2e08a006ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.979524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.979406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.979831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.979810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.979983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.979963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.980311 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.980292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2583030-8f67-49b4-8b2b-5f2e08a006ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:05.982334 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:05.982314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzk6\" (UniqueName: \"kubernetes.io/projected/d2583030-8f67-49b4-8b2b-5f2e08a006ed-kube-api-access-pgzk6\") pod \"prometheus-k8s-0\" (UID: \"d2583030-8f67-49b4-8b2b-5f2e08a006ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:06.002278 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:06.002256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:06.279007 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:06.278975    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 07:07:06.281417 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:07:06.281389    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2583030_8f67_49b4_8b2b_5f2e08a006ed.slice/crio-9d7d0af87842b722c4d702c5db5e391df0a89c437ba6f43504f617481d0a7e0a WatchSource:0}: Error finding container 9d7d0af87842b722c4d702c5db5e391df0a89c437ba6f43504f617481d0a7e0a: Status 404 returned error can't find the container with id 9d7d0af87842b722c4d702c5db5e391df0a89c437ba6f43504f617481d0a7e0a
Sep 08 07:07:06.638607 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:06.638568    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2583030-8f67-49b4-8b2b-5f2e08a006ed" containerID="8cbac7e9494af50bfd9960c7f97bb746de510c01c952d985fc17b14ebdf471a6" exitCode=0
Sep 08 07:07:06.638747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:06.638633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2583030-8f67-49b4-8b2b-5f2e08a006ed","Type":"ContainerDied","Data":"8cbac7e9494af50bfd9960c7f97bb746de510c01c952d985fc17b14ebdf471a6"}
Sep 08 07:07:06.638747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:06.638698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2583030-8f67-49b4-8b2b-5f2e08a006ed","Type":"ContainerStarted","Data":"9d7d0af87842b722c4d702c5db5e391df0a89c437ba6f43504f617481d0a7e0a"}
Sep 08 07:07:06.939814 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:06.939785    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0405bda-7414-4f10-a66e-82b6bf7c199f" path="/var/lib/kubelet/pods/f0405bda-7414-4f10-a66e-82b6bf7c199f/volumes"
Sep 08 07:07:07.646843 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:07.646809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2583030-8f67-49b4-8b2b-5f2e08a006ed","Type":"ContainerStarted","Data":"9e75b8c8ddee634950fa98fe0fbbfd8371a72310731264374fd03167c27af626"}
Sep 08 07:07:07.647037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:07.646873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2583030-8f67-49b4-8b2b-5f2e08a006ed","Type":"ContainerStarted","Data":"757ecfa3db2b4ff4af5eb1351076fd524f7506e7f025feb94a04d143bec4226a"}
Sep 08 07:07:07.647037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:07.646899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2583030-8f67-49b4-8b2b-5f2e08a006ed","Type":"ContainerStarted","Data":"360345c47a7d0caa43e41d9944a3561c39d2ebb942b572afd167595e139df19c"}
Sep 08 07:07:07.647037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:07.646923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2583030-8f67-49b4-8b2b-5f2e08a006ed","Type":"ContainerStarted","Data":"0f0b09c1c54164925b8342726c66fd82786437c428142b00188e034ae7ad4652"}
Sep 08 07:07:07.647037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:07.646947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2583030-8f67-49b4-8b2b-5f2e08a006ed","Type":"ContainerStarted","Data":"d3cd01844722eeccaab023422c90aa5ea5952078b07891031c44125a5fe6882d"}
Sep 08 07:07:07.647037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:07.646972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2583030-8f67-49b4-8b2b-5f2e08a006ed","Type":"ContainerStarted","Data":"a1828422d13e0753a338539e0b67f2ca732f741f0ba30c35bf013914137340cb"}
Sep 08 07:07:07.673186 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:07.673136    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.65323369 podStartE2EDuration="2.673121269s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.887579ms" isStatefulPod=false podCreationTimestamp="2026-09-08 07:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:07:07.67087639 +0000 UTC m=+143.354336543" watchObservedRunningTime="2026-09-08 07:07:07.673121269 +0000 UTC m=+143.356581421"
Sep 08 07:07:08.680180 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:08.680140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:07:08.741998 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:08.741958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:07:11.002769 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:11.002733    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:07:19.626550 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:19.626515    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b5fb64bf-l5qgd"]
Sep 08 07:07:19.729534 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:19.729502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:07:19.794339 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:19.794309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:07:19.847886 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:19.847856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:07:44.651640 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.651534    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85b5fb64bf-l5qgd" podUID="6f9b26b9-e055-4604-9419-eced7bfe2ebc" containerName="console" containerID="cri-o://07e8438d3b4ccd5a6ec99b1b8bd9badac0df7e404c485f704b1749f9dea7db80" gracePeriod=15
Sep 08 07:07:44.783432 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.783398    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f9b26b9-e055-4604-9419-eced7bfe2ebc" containerID="07e8438d3b4ccd5a6ec99b1b8bd9badac0df7e404c485f704b1749f9dea7db80" exitCode=0
Sep 08 07:07:44.783580 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.783455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b5fb64bf-l5qgd" event={"ID":"6f9b26b9-e055-4604-9419-eced7bfe2ebc","Type":"ContainerDied","Data":"07e8438d3b4ccd5a6ec99b1b8bd9badac0df7e404c485f704b1749f9dea7db80"}
Sep 08 07:07:44.886161 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.886138    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:07:44.980738 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.980663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-config\") pod \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") "
Sep 08 07:07:44.980738 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.980700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-oauth-serving-cert\") pod \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") "
Sep 08 07:07:44.980738 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.980727    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-serving-cert\") pod \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") "
Sep 08 07:07:44.981020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.980763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrl9w\" (UniqueName: \"kubernetes.io/projected/6f9b26b9-e055-4604-9419-eced7bfe2ebc-kube-api-access-xrl9w\") pod \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") "
Sep 08 07:07:44.981020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.980779    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-trusted-ca-bundle\") pod \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") "
Sep 08 07:07:44.981020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.980802    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-oauth-config\") pod \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") "
Sep 08 07:07:44.981020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.980840    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-service-ca\") pod \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\" (UID: \"6f9b26b9-e055-4604-9419-eced7bfe2ebc\") "
Sep 08 07:07:44.981219 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.981087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6f9b26b9-e055-4604-9419-eced7bfe2ebc" (UID: "6f9b26b9-e055-4604-9419-eced7bfe2ebc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:44.981284 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.981258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-config" (OuterVolumeSpecName: "console-config") pod "6f9b26b9-e055-4604-9419-eced7bfe2ebc" (UID: "6f9b26b9-e055-4604-9419-eced7bfe2ebc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:44.981370 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.981349    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6f9b26b9-e055-4604-9419-eced7bfe2ebc" (UID: "6f9b26b9-e055-4604-9419-eced7bfe2ebc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:44.981427 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.981344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-service-ca" (OuterVolumeSpecName: "service-ca") pod "6f9b26b9-e055-4604-9419-eced7bfe2ebc" (UID: "6f9b26b9-e055-4604-9419-eced7bfe2ebc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:07:44.983156 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.983128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6f9b26b9-e055-4604-9419-eced7bfe2ebc" (UID: "6f9b26b9-e055-4604-9419-eced7bfe2ebc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:44.983267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.983173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9b26b9-e055-4604-9419-eced7bfe2ebc-kube-api-access-xrl9w" (OuterVolumeSpecName: "kube-api-access-xrl9w") pod "6f9b26b9-e055-4604-9419-eced7bfe2ebc" (UID: "6f9b26b9-e055-4604-9419-eced7bfe2ebc"). InnerVolumeSpecName "kube-api-access-xrl9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:07:44.983267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:44.983173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6f9b26b9-e055-4604-9419-eced7bfe2ebc" (UID: "6f9b26b9-e055-4604-9419-eced7bfe2ebc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:07:45.081723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.081684    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-config\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:45.081723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.081716    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-oauth-serving-cert\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:45.081723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.081726    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-serving-cert\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:45.081723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.081734    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrl9w\" (UniqueName: \"kubernetes.io/projected/6f9b26b9-e055-4604-9419-eced7bfe2ebc-kube-api-access-xrl9w\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:45.081985 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.081743    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-trusted-ca-bundle\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:45.081985 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.081751    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f9b26b9-e055-4604-9419-eced7bfe2ebc-console-oauth-config\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:45.081985 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.081759    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b26b9-e055-4604-9419-eced7bfe2ebc-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:07:45.788624 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.788598    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b5fb64bf-l5qgd"
Sep 08 07:07:45.788624 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.788616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b5fb64bf-l5qgd" event={"ID":"6f9b26b9-e055-4604-9419-eced7bfe2ebc","Type":"ContainerDied","Data":"5cf8fea1aff7d7d8bbdf33446273a9250b22427847bfb8c3c4d0e7e901ebe471"}
Sep 08 07:07:45.789131 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.788667    2580 scope.go:117] "RemoveContainer" containerID="07e8438d3b4ccd5a6ec99b1b8bd9badac0df7e404c485f704b1749f9dea7db80"
Sep 08 07:07:45.810490 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.810443    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b5fb64bf-l5qgd"]
Sep 08 07:07:45.812806 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:45.812783    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85b5fb64bf-l5qgd"]
Sep 08 07:07:46.939255 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:46.939218    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f9b26b9-e055-4604-9419-eced7bfe2ebc" path="/var/lib/kubelet/pods/6f9b26b9-e055-4604-9419-eced7bfe2ebc/volumes"
Sep 08 07:07:59.453743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:59.453710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:07:59.554912 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:07:59.554878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:06.002845 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:06.002811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:08:06.018021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:06.017997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:08:06.877510 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:06.877478    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 07:08:14.739913 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:14.739878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:14.852622 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:14.852587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:18.433029 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:18.432992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:18.498136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:18.498102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:18.577950 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:18.577916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:19.194792 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:19.194759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:19.340554 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:19.340521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:26.662991 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:26.662952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:26.763808 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:26.763772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:29.018186 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:29.018151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:29.115855 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:29.115825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:34.635096 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:34.635057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:34.733714 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:34.733681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:08:57.205826 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:08:57.205792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:18.935948 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:18.935872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:20.427063 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:20.427034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:24.321808 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:24.321770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:24.374443 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:24.374408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:40.547692 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:40.547652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:41.349569 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:41.349531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:44.170828 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:44.170793    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:44.237024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:44.236993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:44.709442 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:44.709418    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 07:09:45.546696 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:45.546656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:45.704359 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:45.704328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:46.365823 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:46.365784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:46.484449 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:46.484417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:48.372784 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:48.372754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:48.781059 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:48.780982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:50.952546 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:50.952511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:51.006546 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:51.006517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:54.650010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.649974    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c8kfb"]
Sep 08 07:09:54.650387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.650234    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f9b26b9-e055-4604-9419-eced7bfe2ebc" containerName="console"
Sep 08 07:09:54.650387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.650244    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f9b26b9-e055-4604-9419-eced7bfe2ebc" containerName="console"
Sep 08 07:09:54.650387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.650306    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f9b26b9-e055-4604-9419-eced7bfe2ebc" containerName="console"
Sep 08 07:09:54.653037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.653021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:54.656424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.656396    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 08 07:09:54.656982 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.656965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 08 07:09:54.657039 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.657016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-95bg2\""
Sep 08 07:09:54.662423 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.662397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c8kfb"]
Sep 08 07:09:54.757551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.757516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfa0d689-8f08-43e0-a3ee-82d9b77105ab-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c8kfb\" (UID: \"bfa0d689-8f08-43e0-a3ee-82d9b77105ab\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:54.757724 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.757574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d284\" (UniqueName: \"kubernetes.io/projected/bfa0d689-8f08-43e0-a3ee-82d9b77105ab-kube-api-access-8d284\") pod \"cert-manager-webhook-6486ffd997-c8kfb\" (UID: \"bfa0d689-8f08-43e0-a3ee-82d9b77105ab\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:54.857919 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.857888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d284\" (UniqueName: \"kubernetes.io/projected/bfa0d689-8f08-43e0-a3ee-82d9b77105ab-kube-api-access-8d284\") pod \"cert-manager-webhook-6486ffd997-c8kfb\" (UID: \"bfa0d689-8f08-43e0-a3ee-82d9b77105ab\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:54.858082 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.857952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfa0d689-8f08-43e0-a3ee-82d9b77105ab-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c8kfb\" (UID: \"bfa0d689-8f08-43e0-a3ee-82d9b77105ab\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:54.865626 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.865593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfa0d689-8f08-43e0-a3ee-82d9b77105ab-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c8kfb\" (UID: \"bfa0d689-8f08-43e0-a3ee-82d9b77105ab\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:54.865752 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.865599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d284\" (UniqueName: \"kubernetes.io/projected/bfa0d689-8f08-43e0-a3ee-82d9b77105ab-kube-api-access-8d284\") pod \"cert-manager-webhook-6486ffd997-c8kfb\" (UID: \"bfa0d689-8f08-43e0-a3ee-82d9b77105ab\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:54.977902 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:54.977820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:55.249803 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:55.249719    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c8kfb"]
Sep 08 07:09:55.252982 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:09:55.252955    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfa0d689_8f08_43e0_a3ee_82d9b77105ab.slice/crio-e3aa2426cd81b15e150284c5f2914f70bfc967dda31c8f7c34a9af86a3c39e60 WatchSource:0}: Error finding container e3aa2426cd81b15e150284c5f2914f70bfc967dda31c8f7c34a9af86a3c39e60: Status 404 returned error can't find the container with id e3aa2426cd81b15e150284c5f2914f70bfc967dda31c8f7c34a9af86a3c39e60
Sep 08 07:09:55.254814 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:55.254795    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 07:09:55.582222 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:55.582183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:55.651810 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:55.651780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:56.247085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:56.247047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb" event={"ID":"bfa0d689-8f08-43e0-a3ee-82d9b77105ab","Type":"ContainerStarted","Data":"e3aa2426cd81b15e150284c5f2914f70bfc967dda31c8f7c34a9af86a3c39e60"}
Sep 08 07:09:57.021772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:57.021732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:09:58.256993 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:58.256953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb" event={"ID":"bfa0d689-8f08-43e0-a3ee-82d9b77105ab","Type":"ContainerStarted","Data":"ba8e879b30b7d846178ad65eb7c278ff4a75a854bff768c5f83ad96ce7002bfe"}
Sep 08 07:09:58.257354 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:58.257035    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:09:58.276036 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:09:58.275986    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb" podStartSLOduration=1.35844199 podStartE2EDuration="4.275969179s" totalImagesPullingTime="2.917527189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:09:58.274217892 +0000 UTC m=+313.957678046" watchObservedRunningTime="2026-09-08 07:09:58.275969179 +0000 UTC m=+313.959429332"
Sep 08 07:10:04.264707 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:04.264664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c8kfb"
Sep 08 07:10:07.738066 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:07.738032    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:07.835803 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:07.835765    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:13.969747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:13.969711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:14.069703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:14.069669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:14.823688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:14.823651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:14.886876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:14.886845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:16.045213 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.045179    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"]
Sep 08 07:10:16.051093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.051068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"
Sep 08 07:10:16.053548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.053525    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 07:10:16.054201 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.054179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2gzrj\""
Sep 08 07:10:16.054201 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.054192    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 07:10:16.056347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.056321    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"]
Sep 08 07:10:16.116564 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.116524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6cc4024-37dd-4bd0-a844-b88bd7f6999a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n2jdc\" (UID: \"c6cc4024-37dd-4bd0-a844-b88bd7f6999a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"
Sep 08 07:10:16.116738 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.116602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7876z\" (UniqueName: \"kubernetes.io/projected/c6cc4024-37dd-4bd0-a844-b88bd7f6999a-kube-api-access-7876z\") pod \"openshift-lws-operator-fd8ccff4c-n2jdc\" (UID: \"c6cc4024-37dd-4bd0-a844-b88bd7f6999a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"
Sep 08 07:10:16.217359 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.217319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7876z\" (UniqueName: \"kubernetes.io/projected/c6cc4024-37dd-4bd0-a844-b88bd7f6999a-kube-api-access-7876z\") pod \"openshift-lws-operator-fd8ccff4c-n2jdc\" (UID: \"c6cc4024-37dd-4bd0-a844-b88bd7f6999a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"
Sep 08 07:10:16.217561 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.217389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6cc4024-37dd-4bd0-a844-b88bd7f6999a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n2jdc\" (UID: \"c6cc4024-37dd-4bd0-a844-b88bd7f6999a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"
Sep 08 07:10:16.217794 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.217769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6cc4024-37dd-4bd0-a844-b88bd7f6999a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n2jdc\" (UID: \"c6cc4024-37dd-4bd0-a844-b88bd7f6999a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"
Sep 08 07:10:16.225946 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.225922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7876z\" (UniqueName: \"kubernetes.io/projected/c6cc4024-37dd-4bd0-a844-b88bd7f6999a-kube-api-access-7876z\") pod \"openshift-lws-operator-fd8ccff4c-n2jdc\" (UID: \"c6cc4024-37dd-4bd0-a844-b88bd7f6999a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"
Sep 08 07:10:16.360853 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.360819    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"
Sep 08 07:10:16.637203 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:16.637126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc"]
Sep 08 07:10:16.641116 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:10:16.641084    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6cc4024_37dd_4bd0_a844_b88bd7f6999a.slice/crio-d6efb835a4c487dde41fd123d0a8d518da931ab8196f7f09fb4ab833a00a35ad WatchSource:0}: Error finding container d6efb835a4c487dde41fd123d0a8d518da931ab8196f7f09fb4ab833a00a35ad: Status 404 returned error can't find the container with id d6efb835a4c487dde41fd123d0a8d518da931ab8196f7f09fb4ab833a00a35ad
Sep 08 07:10:17.322951 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:17.322915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc" event={"ID":"c6cc4024-37dd-4bd0-a844-b88bd7f6999a","Type":"ContainerStarted","Data":"d6efb835a4c487dde41fd123d0a8d518da931ab8196f7f09fb4ab833a00a35ad"}
Sep 08 07:10:20.336805 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:20.336749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc" event={"ID":"c6cc4024-37dd-4bd0-a844-b88bd7f6999a","Type":"ContainerStarted","Data":"fd8d20815810e8d11f3be76b13a4cb9b3377216402acc67cb45c21c2cf42ca80"}
Sep 08 07:10:20.355061 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:20.355004    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n2jdc" podStartSLOduration=1.459605987 podStartE2EDuration="4.354984512s" totalImagesPullingTime="2.895378525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:10:20.353951718 +0000 UTC m=+336.037411872" watchObservedRunningTime="2026-09-08 07:10:20.354984512 +0000 UTC m=+336.038444665"
Sep 08 07:10:23.226800 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:23.226759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:26.046555 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:26.046521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:26.146651 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:26.146613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:27.442246 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.442208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:27.491731 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.491693    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"]
Sep 08 07:10:27.494932 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.494913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.497136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.497113    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 07:10:27.497851 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.497825    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 07:10:27.497991 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.497869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wvhpm\""
Sep 08 07:10:27.497991 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.497896    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 07:10:27.503950 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.503926    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"]
Sep 08 07:10:27.516035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.516011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:27.615424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.615381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-manager-config\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.615617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.615437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpbb\" (UniqueName: \"kubernetes.io/projected/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-kube-api-access-fmpbb\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.615617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.615544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-cert\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.615617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.615573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-metrics-cert\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.716002 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.715918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-cert\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.716002 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.715962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-metrics-cert\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.716002 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.715996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-manager-config\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.716211 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.716026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpbb\" (UniqueName: \"kubernetes.io/projected/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-kube-api-access-fmpbb\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.716786 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.716762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-manager-config\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.718449 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.718418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-metrics-cert\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.718621 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.718602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-cert\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.725454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.725434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpbb\" (UniqueName: \"kubernetes.io/projected/5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d-kube-api-access-fmpbb\") pod \"lws-controller-manager-767bfdf788-qhfpp\" (UID: \"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d\") " pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:27.805517 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:27.805447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:28.079045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:28.079010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"]
Sep 08 07:10:28.082276 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:10:28.082246    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9acb70_e8f3_4f55_aa22_6e16b3b0ac0d.slice/crio-b1e16345fdd1b726e9df862cdb5ca4c7515c2466f7ebd42f7c0d55c1ab0de716 WatchSource:0}: Error finding container b1e16345fdd1b726e9df862cdb5ca4c7515c2466f7ebd42f7c0d55c1ab0de716: Status 404 returned error can't find the container with id b1e16345fdd1b726e9df862cdb5ca4c7515c2466f7ebd42f7c0d55c1ab0de716
Sep 08 07:10:28.368642 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:28.368550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp" event={"ID":"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d","Type":"ContainerStarted","Data":"b1e16345fdd1b726e9df862cdb5ca4c7515c2466f7ebd42f7c0d55c1ab0de716"}
Sep 08 07:10:30.377827 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.377790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp" event={"ID":"5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d","Type":"ContainerStarted","Data":"4d36621475b56f2c313d1127bae97d28a88be94339d5ff3e17c10cd1391669f0"}
Sep 08 07:10:30.378205 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.377913    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:30.394542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.394485    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp" podStartSLOduration=1.7901925319999998 podStartE2EDuration="3.394442437s" totalImagesPullingTime="1.604249905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:10:30.393342251 +0000 UTC m=+346.076802406" watchObservedRunningTime="2026-09-08 07:10:30.394442437 +0000 UTC m=+346.077902654"
Sep 08 07:10:30.728893 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.728814    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-768697c4b8-wdwdt"]
Sep 08 07:10:30.732114 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.732097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.734151 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.734129    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 07:10:30.734269 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.734233    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 07:10:30.734774 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.734758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mxlnj\""
Sep 08 07:10:30.734852 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.734796    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 07:10:30.734908 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.734864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 07:10:30.734908 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.734881    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 07:10:30.734908 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.734881    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 07:10:30.735061 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.734864    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 07:10:30.739239 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.738997    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 07:10:30.741547 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.741526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768697c4b8-wdwdt"]
Sep 08 07:10:30.784205 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.784173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:30.836050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.836017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:30.846158 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.846134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-service-ca\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.846275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.846182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-serving-cert\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.846275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.846224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-config\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.846390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.846282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-trusted-ca-bundle\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.846390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.846313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-oauth-serving-cert\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.846390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.846349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-oauth-config\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.846390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.846384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngq5\" (UniqueName: \"kubernetes.io/projected/8f4df6fa-46c4-406b-9f2e-bf53c32233da-kube-api-access-zngq5\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.947536 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.947500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngq5\" (UniqueName: \"kubernetes.io/projected/8f4df6fa-46c4-406b-9f2e-bf53c32233da-kube-api-access-zngq5\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.947700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.947553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-service-ca\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.947700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.947649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-serving-cert\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.947700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.947677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-config\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.947700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.947697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-trusted-ca-bundle\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.947887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.947722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-oauth-serving-cert\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.947887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.947757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-oauth-config\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.948547 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.948520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-config\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.948722 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.948575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-oauth-serving-cert\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.948722 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.948632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-service-ca\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.948802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.948739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4df6fa-46c4-406b-9f2e-bf53c32233da-trusted-ca-bundle\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.950272 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.950248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-oauth-config\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.950429 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.950411    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4df6fa-46c4-406b-9f2e-bf53c32233da-console-serving-cert\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:30.954892 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:30.954870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngq5\" (UniqueName: \"kubernetes.io/projected/8f4df6fa-46c4-406b-9f2e-bf53c32233da-kube-api-access-zngq5\") pod \"console-768697c4b8-wdwdt\" (UID: \"8f4df6fa-46c4-406b-9f2e-bf53c32233da\") " pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:31.042291 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:31.042206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:31.317992 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:31.317905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768697c4b8-wdwdt"]
Sep 08 07:10:31.321024 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:10:31.320983    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4df6fa_46c4_406b_9f2e_bf53c32233da.slice/crio-63411edaf3a3550525f0e8e977ff216cf8f26805fdc372d198be6cd025c002c5 WatchSource:0}: Error finding container 63411edaf3a3550525f0e8e977ff216cf8f26805fdc372d198be6cd025c002c5: Status 404 returned error can't find the container with id 63411edaf3a3550525f0e8e977ff216cf8f26805fdc372d198be6cd025c002c5
Sep 08 07:10:31.384700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:31.384667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768697c4b8-wdwdt" event={"ID":"8f4df6fa-46c4-406b-9f2e-bf53c32233da","Type":"ContainerStarted","Data":"63411edaf3a3550525f0e8e977ff216cf8f26805fdc372d198be6cd025c002c5"}
Sep 08 07:10:32.390187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:32.390149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768697c4b8-wdwdt" event={"ID":"8f4df6fa-46c4-406b-9f2e-bf53c32233da","Type":"ContainerStarted","Data":"ace7973e2cd8557c054552b62d183b8cc5446e1db881d3cac6b0415b04bcb6dd"}
Sep 08 07:10:32.408273 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:32.408220    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-768697c4b8-wdwdt" podStartSLOduration=2.408204587 podStartE2EDuration="2.408204587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:10:32.406930174 +0000 UTC m=+348.090390339" watchObservedRunningTime="2026-09-08 07:10:32.408204587 +0000 UTC m=+348.091664747"
Sep 08 07:10:41.043036 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:41.042933    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:41.043036 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:41.043008    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:41.047979 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:41.047950    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:41.387750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:41.387691    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-767bfdf788-qhfpp"
Sep 08 07:10:41.427769 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:41.427738    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-768697c4b8-wdwdt"
Sep 08 07:10:41.584822 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:41.584789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:41.682717 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:41.682638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:41.780512 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:41.780478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:43.755654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:43.755617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:43.855788 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:43.855748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:44.380661 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.380629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:44.654210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.654121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:44.757104 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.757069    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:44.784777 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.784742    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"]
Sep 08 07:10:44.786923 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.786906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:44.789068 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.789025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 07:10:44.789068 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.789022    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 07:10:44.789256 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.789166    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nw77s\""
Sep 08 07:10:44.789326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.789267    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 07:10:44.789326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.789304    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 07:10:44.798736 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.798708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"]
Sep 08 07:10:44.961741 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.961658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:44.973928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.973886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/557dcd52-2243-4aef-8041-1fcb15748061-tmp\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:44.974027 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.973941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/557dcd52-2243-4aef-8041-1fcb15748061-manifests\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:44.974027 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.974010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcnd\" (UniqueName: \"kubernetes.io/projected/557dcd52-2243-4aef-8041-1fcb15748061-kube-api-access-xqcnd\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:44.974096 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.974054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/557dcd52-2243-4aef-8041-1fcb15748061-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:44.974096 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:44.974073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/557dcd52-2243-4aef-8041-1fcb15748061-webhook-cert\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.066609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.066563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:45.074804 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.074758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/557dcd52-2243-4aef-8041-1fcb15748061-tmp\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.074878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.074816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/557dcd52-2243-4aef-8041-1fcb15748061-manifests\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.074878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.074841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcnd\" (UniqueName: \"kubernetes.io/projected/557dcd52-2243-4aef-8041-1fcb15748061-kube-api-access-xqcnd\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.074878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.074861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/557dcd52-2243-4aef-8041-1fcb15748061-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.074878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.074876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/557dcd52-2243-4aef-8041-1fcb15748061-webhook-cert\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.075275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.075242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/557dcd52-2243-4aef-8041-1fcb15748061-manifests\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.077148 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.077125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/557dcd52-2243-4aef-8041-1fcb15748061-tmp\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.077350 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.077332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/557dcd52-2243-4aef-8041-1fcb15748061-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.077407 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.077389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/557dcd52-2243-4aef-8041-1fcb15748061-webhook-cert\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.082749 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.082726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcnd\" (UniqueName: \"kubernetes.io/projected/557dcd52-2243-4aef-8041-1fcb15748061-kube-api-access-xqcnd\") pod \"opendatahub-operator-controller-manager-79c9ffb96f-cqzk9\" (UID: \"557dcd52-2243-4aef-8041-1fcb15748061\") " pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.097602 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.097570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:45.164560 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.164527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:45.262495 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.262380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:45.364427 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.364389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:10:45.373757 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.373732    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"]
Sep 08 07:10:45.375550 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:10:45.375523    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557dcd52_2243_4aef_8041_1fcb15748061.slice/crio-91d661e03c5cd1bb1943a3f6ed557d56527832765dd5121eb17959e1c8edfe99 WatchSource:0}: Error finding container 91d661e03c5cd1bb1943a3f6ed557d56527832765dd5121eb17959e1c8edfe99: Status 404 returned error can't find the container with id 91d661e03c5cd1bb1943a3f6ed557d56527832765dd5121eb17959e1c8edfe99
Sep 08 07:10:45.440503 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:45.440442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9" event={"ID":"557dcd52-2243-4aef-8041-1fcb15748061","Type":"ContainerStarted","Data":"91d661e03c5cd1bb1943a3f6ed557d56527832765dd5121eb17959e1c8edfe99"}
Sep 08 07:10:49.460066 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:49.460028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9" event={"ID":"557dcd52-2243-4aef-8041-1fcb15748061","Type":"ContainerStarted","Data":"2db65b4ee5371d64285d04d3188281846acee41259242fc35c6f3dbc34ae2b72"}
Sep 08 07:10:50.465573 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:50.465539    2580 generic.go:356] "Generic (PLEG): container finished" podID="557dcd52-2243-4aef-8041-1fcb15748061" containerID="2db65b4ee5371d64285d04d3188281846acee41259242fc35c6f3dbc34ae2b72" exitCode=0
Sep 08 07:10:50.465960 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:50.465595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9" event={"ID":"557dcd52-2243-4aef-8041-1fcb15748061","Type":"ContainerDied","Data":"2db65b4ee5371d64285d04d3188281846acee41259242fc35c6f3dbc34ae2b72"}
Sep 08 07:10:51.472022 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:51.471983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9" event={"ID":"557dcd52-2243-4aef-8041-1fcb15748061","Type":"ContainerStarted","Data":"c616903397266d2a6fdf61d5a80ee2d785952fe7bab5267504dc59609c6a6cfc"}
Sep 08 07:10:51.472419 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:51.472091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:10:51.494506 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:10:51.494427    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9" podStartSLOduration=2.787042285 podStartE2EDuration="7.494409572s" totalImagesPullingTime="3.666767719s" totalInitContainerRuntime="1.040599568s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:10:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:10:51.492379434 +0000 UTC m=+367.175839621" watchObservedRunningTime="2026-09-08 07:10:51.494409572 +0000 UTC m=+367.177869724"
Sep 08 07:11:02.479228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:02.479185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79c9ffb96f-cqzk9"
Sep 08 07:11:03.969860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:03.969823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:04.079980 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:04.079945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:09.618151 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.618119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:09.651860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.651827    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cd6874cf-t744g"]
Sep 08 07:11:09.655377 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.655357    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.657263 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.657238    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dtt8k\""
Sep 08 07:11:09.657389 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.657353    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 07:11:09.658053 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.658038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 07:11:09.658145 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.658102    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 07:11:09.658205 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.658137    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 07:11:09.665383 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.665361    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cd6874cf-t744g"]
Sep 08 07:11:09.784199 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.784163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kln2t\" (UniqueName: \"kubernetes.io/projected/c39bc3c3-e554-4582-8949-da95477c5b82-kube-api-access-kln2t\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.784374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.784222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39bc3c3-e554-4582-8949-da95477c5b82-tls-certs\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.784374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.784322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39bc3c3-e554-4582-8949-da95477c5b82-tmp\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.785755 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.785730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:09.885745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.885642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kln2t\" (UniqueName: \"kubernetes.io/projected/c39bc3c3-e554-4582-8949-da95477c5b82-kube-api-access-kln2t\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.885745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.885728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39bc3c3-e554-4582-8949-da95477c5b82-tls-certs\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.885979 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.885784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39bc3c3-e554-4582-8949-da95477c5b82-tmp\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.888000 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.887974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39bc3c3-e554-4582-8949-da95477c5b82-tmp\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.888220 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.888200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39bc3c3-e554-4582-8949-da95477c5b82-tls-certs\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.893816 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.893782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kln2t\" (UniqueName: \"kubernetes.io/projected/c39bc3c3-e554-4582-8949-da95477c5b82-kube-api-access-kln2t\") pod \"kube-auth-proxy-cd6874cf-t744g\" (UID: \"c39bc3c3-e554-4582-8949-da95477c5b82\") " pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:09.964992 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:09.964958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g"
Sep 08 07:11:10.242995 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:10.242911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cd6874cf-t744g"]
Sep 08 07:11:10.246082 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:11:10.246054    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39bc3c3_e554_4582_8949_da95477c5b82.slice/crio-ab560698174acfbb7d8aac6bdc3e758884d5abbacb0a985e8df7bfdd3585dc36 WatchSource:0}: Error finding container ab560698174acfbb7d8aac6bdc3e758884d5abbacb0a985e8df7bfdd3585dc36: Status 404 returned error can't find the container with id ab560698174acfbb7d8aac6bdc3e758884d5abbacb0a985e8df7bfdd3585dc36
Sep 08 07:11:10.549559 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:10.549453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g" event={"ID":"c39bc3c3-e554-4582-8949-da95477c5b82","Type":"ContainerStarted","Data":"ab560698174acfbb7d8aac6bdc3e758884d5abbacb0a985e8df7bfdd3585dc36"}
Sep 08 07:11:10.814619 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:10.814543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:10.880553 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:10.880525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:11.698849 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:11.698814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:13.565447 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:13.565352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g" event={"ID":"c39bc3c3-e554-4582-8949-da95477c5b82","Type":"ContainerStarted","Data":"83c58cf2fcdf1ccaae1f3bd450cc8402158806dd2dbe62c54b3a8b4a6bb2dfca"}
Sep 08 07:11:13.582040 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:13.581988    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cd6874cf-t744g" podStartSLOduration=1.712504226 podStartE2EDuration="4.581972676s" totalImagesPullingTime="2.86946845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:11:13.580157188 +0000 UTC m=+389.263617319" watchObservedRunningTime="2026-09-08 07:11:13.581972676 +0000 UTC m=+389.265432855"
Sep 08 07:11:15.131740 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:15.131708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:15.232120 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:15.232088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:15.334874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:15.334843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:15.432855 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:15.432771    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:15.591397 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:15.591356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:15.930069 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:15.930039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:16.130103 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:16.130072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:16.375515 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:16.375475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:16.551327 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:16.551291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:24.400836 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:24.400797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:24.509739 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:24.509706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:24.659519 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:24.659426    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:24.768481 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:24.768435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:24.908638 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:24.908605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:25.010795 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:25.010718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:28.945956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:28.945925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:11:35.619695 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:11:35.619660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:11.910625 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:11.910543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:11.967439 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:11.967409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:13.873387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:13.873358    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:16.087782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:16.087745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:16.238248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:16.238215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:40.297944 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:40.297898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:42.691870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:42.691833    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:43.095728 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:43.095699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:44.524569 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:44.524540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:45.704800 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:45.704762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:45.927580 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:45.927541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:47.342871 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:47.342838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:47.940637 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:47.940601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:48.153968 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:48.153937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:48.312427 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:48.312339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:48.418821 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:48.418787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:48.728921 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:48.728890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:48.836042 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:48.836013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:53.900740 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:53.900704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:53.998583 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:53.998550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:58.650926 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.650898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:58.657298 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.657264    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"]
Sep 08 07:12:58.660391 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.660374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:58.662510 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.662491    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 07:12:58.662642 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.662511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rgb29\""
Sep 08 07:12:58.662642 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.662627    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 07:12:58.662950 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.662935    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 07:12:58.663033 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.662936    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 07:12:58.666693 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.666672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"]
Sep 08 07:12:58.769570 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.769524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/920e6331-fed7-4317-b3a6-05f6bd6032a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:58.769753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.769662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/920e6331-fed7-4317-b3a6-05f6bd6032a2-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:58.769753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.769731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qcr\" (UniqueName: \"kubernetes.io/projected/920e6331-fed7-4317-b3a6-05f6bd6032a2-kube-api-access-z9qcr\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:58.870841 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.870804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/920e6331-fed7-4317-b3a6-05f6bd6032a2-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:58.871035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.870862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qcr\" (UniqueName: \"kubernetes.io/projected/920e6331-fed7-4317-b3a6-05f6bd6032a2-kube-api-access-z9qcr\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:58.871035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.870901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/920e6331-fed7-4317-b3a6-05f6bd6032a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:58.871035 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:12:58.871018    2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 08 07:12:58.871205 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:12:58.871078    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920e6331-fed7-4317-b3a6-05f6bd6032a2-plugin-serving-cert podName:920e6331-fed7-4317-b3a6-05f6bd6032a2 nodeName:}" failed. No retries permitted until 2026-09-08 07:12:59.371057636 +0000 UTC m=+495.054517767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/920e6331-fed7-4317-b3a6-05f6bd6032a2-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-pg7jc" (UID: "920e6331-fed7-4317-b3a6-05f6bd6032a2") : secret "plugin-serving-cert" not found
Sep 08 07:12:58.871552 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.871532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/920e6331-fed7-4317-b3a6-05f6bd6032a2-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:58.879130 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:58.879110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qcr\" (UniqueName: \"kubernetes.io/projected/920e6331-fed7-4317-b3a6-05f6bd6032a2-kube-api-access-z9qcr\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:59.124975 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:59.124940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:59.375753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:59.375660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/920e6331-fed7-4317-b3a6-05f6bd6032a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:59.378074 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:59.378052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/920e6331-fed7-4317-b3a6-05f6bd6032a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-pg7jc\" (UID: \"920e6331-fed7-4317-b3a6-05f6bd6032a2\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:59.503652 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:59.503621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:59.570263 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:59.570219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"
Sep 08 07:12:59.603750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:59.603721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:12:59.842734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:59.842703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc"]
Sep 08 07:12:59.846136 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:12:59.846107    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod920e6331_fed7_4317_b3a6_05f6bd6032a2.slice/crio-7c13b39f8054cc54423bcb46a398052db83c25248f073750ed29a37a37608c3b WatchSource:0}: Error finding container 7c13b39f8054cc54423bcb46a398052db83c25248f073750ed29a37a37608c3b: Status 404 returned error can't find the container with id 7c13b39f8054cc54423bcb46a398052db83c25248f073750ed29a37a37608c3b
Sep 08 07:12:59.931905 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:12:59.931877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:00.003857 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:00.003824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc" event={"ID":"920e6331-fed7-4317-b3a6-05f6bd6032a2","Type":"ContainerStarted","Data":"7c13b39f8054cc54423bcb46a398052db83c25248f073750ed29a37a37608c3b"}
Sep 08 07:13:00.032311 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:00.032279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:01.540667 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:01.540634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:01.642215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:01.642174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:05.032649 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:05.032611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc" event={"ID":"920e6331-fed7-4317-b3a6-05f6bd6032a2","Type":"ContainerStarted","Data":"f90c26f91ea1fb10881ace735da11a66c9cfec02d86f6f3a7575f68390b66ffb"}
Sep 08 07:13:05.049526 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:05.049468    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-pg7jc" podStartSLOduration=2.557766155 podStartE2EDuration="7.049437555s" totalImagesPullingTime="4.4916714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:13:05.048175413 +0000 UTC m=+500.731635547" watchObservedRunningTime="2026-09-08 07:13:05.049437555 +0000 UTC m=+500.732897707"
Sep 08 07:13:06.265990 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:06.265952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:06.361676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:06.361638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:06.461597 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:06.461566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:08.938976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:08.938942    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"]
Sep 08 07:13:08.942218 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:08.942201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:08.944293 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:08.944274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bnvtt\""
Sep 08 07:13:08.950340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:08.950317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"]
Sep 08 07:13:09.061515 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.061479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" (UID: \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:09.061686 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.061537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwn78\" (UniqueName: \"kubernetes.io/projected/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-kube-api-access-vwn78\") pod \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" (UID: \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:09.162836 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.162802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" (UID: \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:09.162969 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.162861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwn78\" (UniqueName: \"kubernetes.io/projected/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-kube-api-access-vwn78\") pod \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" (UID: \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:09.163261 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.163236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" (UID: \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:09.171821 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.171794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwn78\" (UniqueName: \"kubernetes.io/projected/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-kube-api-access-vwn78\") pod \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" (UID: \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:09.252395 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.252312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:09.528540 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.528515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"]
Sep 08 07:13:09.531209 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:13:09.531183    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdde8f8_742d_4bd3_ad12_9a53b1850d2e.slice/crio-96c20da580ba0379079e984a18512656ba7699d87ef1184a06da4983ff9fcb6c WatchSource:0}: Error finding container 96c20da580ba0379079e984a18512656ba7699d87ef1184a06da4983ff9fcb6c: Status 404 returned error can't find the container with id 96c20da580ba0379079e984a18512656ba7699d87ef1184a06da4983ff9fcb6c
Sep 08 07:13:09.913727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.913694    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"]
Sep 08 07:13:09.924555 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.924512    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"]
Sep 08 07:13:09.973899 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.973867    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"]
Sep 08 07:13:09.978288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.978261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"
Sep 08 07:13:09.981178 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.981153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d7lnn\""
Sep 08 07:13:09.994238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:09.994209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"]
Sep 08 07:13:10.071655 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:10.071625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4prrp\" (UniqueName: \"kubernetes.io/projected/4a5651b3-8356-48ef-a4fe-759712a192c7-kube-api-access-4prrp\") pod \"limitador-operator-controller-manager-6c549b6d85-8hv5c\" (UID: \"4a5651b3-8356-48ef-a4fe-759712a192c7\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"
Sep 08 07:13:10.172699 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:10.172605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4prrp\" (UniqueName: \"kubernetes.io/projected/4a5651b3-8356-48ef-a4fe-759712a192c7-kube-api-access-4prrp\") pod \"limitador-operator-controller-manager-6c549b6d85-8hv5c\" (UID: \"4a5651b3-8356-48ef-a4fe-759712a192c7\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"
Sep 08 07:13:10.182827 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:10.182792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4prrp\" (UniqueName: \"kubernetes.io/projected/4a5651b3-8356-48ef-a4fe-759712a192c7-kube-api-access-4prrp\") pod \"limitador-operator-controller-manager-6c549b6d85-8hv5c\" (UID: \"4a5651b3-8356-48ef-a4fe-759712a192c7\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"
Sep 08 07:13:10.289201 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:10.288988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"
Sep 08 07:13:10.579569 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:10.579531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"]
Sep 08 07:13:10.584515 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:13:10.584434    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5651b3_8356_48ef_a4fe_759712a192c7.slice/crio-2dc185f2a59d5645459faeed5893c2ebc61fa1346ac206b9dec3dbc76318f9b3 WatchSource:0}: Error finding container 2dc185f2a59d5645459faeed5893c2ebc61fa1346ac206b9dec3dbc76318f9b3: Status 404 returned error can't find the container with id 2dc185f2a59d5645459faeed5893c2ebc61fa1346ac206b9dec3dbc76318f9b3
Sep 08 07:13:11.063499 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:11.063376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c" event={"ID":"4a5651b3-8356-48ef-a4fe-759712a192c7","Type":"ContainerStarted","Data":"2dc185f2a59d5645459faeed5893c2ebc61fa1346ac206b9dec3dbc76318f9b3"}
Sep 08 07:13:12.907717 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:12.907674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:13.063156 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:13.063122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:14.080603 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.080571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c" event={"ID":"4a5651b3-8356-48ef-a4fe-759712a192c7","Type":"ContainerStarted","Data":"2506f3285731bfcd1b08a634ecd4c9294a622307525578b2dd1547810e39b989"}
Sep 08 07:13:14.081040 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.080692    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"
Sep 08 07:13:14.082296 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.082270    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6" podUID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" containerName="manager" containerID="cri-o://87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce" gracePeriod=2
Sep 08 07:13:14.097745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.097704    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c" podStartSLOduration=2.0999928900000002 podStartE2EDuration="5.097688251s" totalImagesPullingTime="2.997695361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:13:14.096782103 +0000 UTC m=+509.780242256" watchObservedRunningTime="2026-09-08 07:13:14.097688251 +0000 UTC m=+509.781148404"
Sep 08 07:13:14.317073 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.317043    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:14.318727 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:13:14.318701    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" is forbidden: User \"system:node:ip-10-0-139-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-109.ec2.internal' and this object" podUID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:14.413291 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.413214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwn78\" (UniqueName: \"kubernetes.io/projected/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-kube-api-access-vwn78\") pod \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\" (UID: \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\") "
Sep 08 07:13:14.413430 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.413350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-extensions-socket-volume\") pod \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\" (UID: \"9cdde8f8-742d-4bd3-ad12-9a53b1850d2e\") "
Sep 08 07:13:14.413606 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.413582    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" (UID: "9cdde8f8-742d-4bd3-ad12-9a53b1850d2e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:13:14.415286 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.415267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-kube-api-access-vwn78" (OuterVolumeSpecName: "kube-api-access-vwn78") pod "9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" (UID: "9cdde8f8-742d-4bd3-ad12-9a53b1850d2e"). InnerVolumeSpecName "kube-api-access-vwn78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:13:14.514359 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.514329    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-extensions-socket-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:13:14.514359 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.514354    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwn78\" (UniqueName: \"kubernetes.io/projected/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e-kube-api-access-vwn78\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:13:14.938263 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:13:14.938212    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" is forbidden: User \"system:node:ip-10-0-139-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-109.ec2.internal' and this object" podUID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:14.941092 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:14.941068    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" path="/var/lib/kubelet/pods/9cdde8f8-742d-4bd3-ad12-9a53b1850d2e/volumes"
Sep 08 07:13:15.087426 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:15.087393    2580 generic.go:356] "Generic (PLEG): container finished" podID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" containerID="87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce" exitCode=2
Sep 08 07:13:15.087858 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:15.087438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:15.087858 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:15.087511    2580 scope.go:117] "RemoveContainer" containerID="87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce"
Sep 08 07:13:15.093103 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:13:15.093049    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-zzlm6\" is forbidden: User \"system:node:ip-10-0-139-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-109.ec2.internal' and this object" podUID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-zzlm6"
Sep 08 07:13:15.101138 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:15.101105    2580 scope.go:117] "RemoveContainer" containerID="87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce"
Sep 08 07:13:15.101547 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:13:15.101520    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce\": container with ID starting with 87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce not found: ID does not exist" containerID="87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce"
Sep 08 07:13:15.101632 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:15.101576    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce"} err="failed to get container status \"87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce\": rpc error: code = NotFound desc = could not find container \"87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce\": container with ID starting with 87c45c2cb4d6a5eac04a40fcdf17b6d2d3772ce8827f1e15efcd95c8f1af6cce not found: ID does not exist"
Sep 08 07:13:18.182088 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:18.182058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:18.239471 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:18.239427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:18.302825 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:18.302793    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:18.354495 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:18.354453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:18.704445 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:18.704414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:18.805413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:18.805380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:22.126659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:22.126631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:22.227527 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:22.227498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:25.090036 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:25.090009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-8hv5c"
Sep 08 07:13:25.863676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:25.863641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:25.963556 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:25.963527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:26.064678 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:26.064647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:38.380064 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:38.379985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:38.590709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:38.590675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:49.150362 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.150329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:49.254091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.254058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:49.312339 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.312294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:49.986066 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.986027    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-spr2l"]
Sep 08 07:13:49.986379 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.986366    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" containerName="manager"
Sep 08 07:13:49.986424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.986381    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" containerName="manager"
Sep 08 07:13:49.986481 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.986468    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cdde8f8-742d-4bd3-ad12-9a53b1850d2e" containerName="manager"
Sep 08 07:13:49.988946 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.988922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-spr2l"
Sep 08 07:13:49.990879 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.990859    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dmz6h\""
Sep 08 07:13:49.996199 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:49.995912    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-spr2l"]
Sep 08 07:13:50.041772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.041740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:50.110883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.110848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx7sg\" (UniqueName: \"kubernetes.io/projected/e9afa55a-8fb8-4995-904d-c68643a96f73-kube-api-access-rx7sg\") pod \"authorino-b54df94b4-spr2l\" (UID: \"e9afa55a-8fb8-4995-904d-c68643a96f73\") " pod="kuadrant-system/authorino-b54df94b4-spr2l"
Sep 08 07:13:50.117235 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.117202    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-v42mn"]
Sep 08 07:13:50.118375 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.118356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:13:50.119572 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.119557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-v42mn"
Sep 08 07:13:50.124391 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.124364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-v42mn"]
Sep 08 07:13:50.211275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.211234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx7sg\" (UniqueName: \"kubernetes.io/projected/e9afa55a-8fb8-4995-904d-c68643a96f73-kube-api-access-rx7sg\") pod \"authorino-b54df94b4-spr2l\" (UID: \"e9afa55a-8fb8-4995-904d-c68643a96f73\") " pod="kuadrant-system/authorino-b54df94b4-spr2l"
Sep 08 07:13:50.211722 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.211388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb52b\" (UniqueName: \"kubernetes.io/projected/2ea560ec-12ce-4998-8d8d-6ff3e65835dc-kube-api-access-tb52b\") pod \"authorino-c548b5d7d-v42mn\" (UID: \"2ea560ec-12ce-4998-8d8d-6ff3e65835dc\") " pod="kuadrant-system/authorino-c548b5d7d-v42mn"
Sep 08 07:13:50.219118 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.219091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx7sg\" (UniqueName: \"kubernetes.io/projected/e9afa55a-8fb8-4995-904d-c68643a96f73-kube-api-access-rx7sg\") pod \"authorino-b54df94b4-spr2l\" (UID: \"e9afa55a-8fb8-4995-904d-c68643a96f73\") " pod="kuadrant-system/authorino-b54df94b4-spr2l"
Sep 08 07:13:50.299610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.299524    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-spr2l"
Sep 08 07:13:50.312171 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.312096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb52b\" (UniqueName: \"kubernetes.io/projected/2ea560ec-12ce-4998-8d8d-6ff3e65835dc-kube-api-access-tb52b\") pod \"authorino-c548b5d7d-v42mn\" (UID: \"2ea560ec-12ce-4998-8d8d-6ff3e65835dc\") " pod="kuadrant-system/authorino-c548b5d7d-v42mn"
Sep 08 07:13:50.320780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.320592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb52b\" (UniqueName: \"kubernetes.io/projected/2ea560ec-12ce-4998-8d8d-6ff3e65835dc-kube-api-access-tb52b\") pod \"authorino-c548b5d7d-v42mn\" (UID: \"2ea560ec-12ce-4998-8d8d-6ff3e65835dc\") " pod="kuadrant-system/authorino-c548b5d7d-v42mn"
Sep 08 07:13:50.429630 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.429587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-v42mn"
Sep 08 07:13:50.569929 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.569903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-spr2l"]
Sep 08 07:13:50.573083 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:13:50.573058    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9afa55a_8fb8_4995_904d_c68643a96f73.slice/crio-16682311a3bf7068bcd6d2567be49c73a2312ad88c11c745e0c90aed702d9415 WatchSource:0}: Error finding container 16682311a3bf7068bcd6d2567be49c73a2312ad88c11c745e0c90aed702d9415: Status 404 returned error can't find the container with id 16682311a3bf7068bcd6d2567be49c73a2312ad88c11c745e0c90aed702d9415
Sep 08 07:13:50.697687 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:50.697657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-v42mn"]
Sep 08 07:13:50.700441 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:13:50.700409    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea560ec_12ce_4998_8d8d_6ff3e65835dc.slice/crio-7a1284a915f071334bf91e482f71032561ec2725834e36f47343c93ee53bf857 WatchSource:0}: Error finding container 7a1284a915f071334bf91e482f71032561ec2725834e36f47343c93ee53bf857: Status 404 returned error can't find the container with id 7a1284a915f071334bf91e482f71032561ec2725834e36f47343c93ee53bf857
Sep 08 07:13:51.245787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:51.245745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-v42mn" event={"ID":"2ea560ec-12ce-4998-8d8d-6ff3e65835dc","Type":"ContainerStarted","Data":"7a1284a915f071334bf91e482f71032561ec2725834e36f47343c93ee53bf857"}
Sep 08 07:13:51.246819 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:51.246788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-spr2l" event={"ID":"e9afa55a-8fb8-4995-904d-c68643a96f73","Type":"ContainerStarted","Data":"16682311a3bf7068bcd6d2567be49c73a2312ad88c11c745e0c90aed702d9415"}
Sep 08 07:13:53.258489 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:53.258439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-v42mn" event={"ID":"2ea560ec-12ce-4998-8d8d-6ff3e65835dc","Type":"ContainerStarted","Data":"7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e"}
Sep 08 07:13:53.259923 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:53.259893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-spr2l" event={"ID":"e9afa55a-8fb8-4995-904d-c68643a96f73","Type":"ContainerStarted","Data":"0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3"}
Sep 08 07:13:53.272988 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:53.272937    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-v42mn" podStartSLOduration=1.264381279 podStartE2EDuration="3.272917844s" totalImagesPullingTime="2.008536565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:13:53.271838783 +0000 UTC m=+548.955298937" watchObservedRunningTime="2026-09-08 07:13:53.272917844 +0000 UTC m=+548.956378060"
Sep 08 07:13:53.286028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:53.285974    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-spr2l" podStartSLOduration=2.16005646 podStartE2EDuration="4.285957917s" totalImagesPullingTime="2.125901457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:13:53.285090031 +0000 UTC m=+548.968550186" watchObservedRunningTime="2026-09-08 07:13:53.285957917 +0000 UTC m=+548.969418069"
Sep 08 07:13:53.303593 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:53.303561    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-spr2l"]
Sep 08 07:13:55.268009 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:55.267961    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-spr2l" podUID="e9afa55a-8fb8-4995-904d-c68643a96f73" containerName="authorino" containerID="cri-o://0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3" gracePeriod=30
Sep 08 07:13:55.519311 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:55.519250    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-spr2l"
Sep 08 07:13:55.561090 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:55.561054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx7sg\" (UniqueName: \"kubernetes.io/projected/e9afa55a-8fb8-4995-904d-c68643a96f73-kube-api-access-rx7sg\") pod \"e9afa55a-8fb8-4995-904d-c68643a96f73\" (UID: \"e9afa55a-8fb8-4995-904d-c68643a96f73\") "
Sep 08 07:13:55.563180 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:55.563154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9afa55a-8fb8-4995-904d-c68643a96f73-kube-api-access-rx7sg" (OuterVolumeSpecName: "kube-api-access-rx7sg") pod "e9afa55a-8fb8-4995-904d-c68643a96f73" (UID: "e9afa55a-8fb8-4995-904d-c68643a96f73"). InnerVolumeSpecName "kube-api-access-rx7sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:13:55.662341 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:55.662288    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx7sg\" (UniqueName: \"kubernetes.io/projected/e9afa55a-8fb8-4995-904d-c68643a96f73-kube-api-access-rx7sg\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:13:56.273902 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.273869    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9afa55a-8fb8-4995-904d-c68643a96f73" containerID="0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3" exitCode=0
Sep 08 07:13:56.274404 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.273909    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-spr2l"
Sep 08 07:13:56.274404 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.273962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-spr2l" event={"ID":"e9afa55a-8fb8-4995-904d-c68643a96f73","Type":"ContainerDied","Data":"0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3"}
Sep 08 07:13:56.274404 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.274001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-spr2l" event={"ID":"e9afa55a-8fb8-4995-904d-c68643a96f73","Type":"ContainerDied","Data":"16682311a3bf7068bcd6d2567be49c73a2312ad88c11c745e0c90aed702d9415"}
Sep 08 07:13:56.274404 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.274019    2580 scope.go:117] "RemoveContainer" containerID="0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3"
Sep 08 07:13:56.282257 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.282237    2580 scope.go:117] "RemoveContainer" containerID="0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3"
Sep 08 07:13:56.282532 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:13:56.282516    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3\": container with ID starting with 0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3 not found: ID does not exist" containerID="0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3"
Sep 08 07:13:56.282588 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.282549    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3"} err="failed to get container status \"0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3\": rpc error: code = NotFound desc = could not find container \"0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3\": container with ID starting with 0f9a41adb7b07f6159a530ee874e72c7f91f7d2681712a8ee9a71002fe4baaa3 not found: ID does not exist"
Sep 08 07:13:56.293931 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.293907    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-spr2l"]
Sep 08 07:13:56.298607 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.298587    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-spr2l"]
Sep 08 07:13:56.940539 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:13:56.940506    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9afa55a-8fb8-4995-904d-c68643a96f73" path="/var/lib/kubelet/pods/e9afa55a-8fb8-4995-904d-c68643a96f73/volumes"
Sep 08 07:14:06.551410 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:06.551366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:06.651032 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:06.650997    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:07.015071 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:07.015035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:19.631370 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.631336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:19.776119 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.776085    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:19.844878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.844832    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-x2zfs"]
Sep 08 07:14:19.845156 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.845143    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9afa55a-8fb8-4995-904d-c68643a96f73" containerName="authorino"
Sep 08 07:14:19.845199 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.845158    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9afa55a-8fb8-4995-904d-c68643a96f73" containerName="authorino"
Sep 08 07:14:19.845233 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.845222    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9afa55a-8fb8-4995-904d-c68643a96f73" containerName="authorino"
Sep 08 07:14:19.847050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.847032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-x2zfs"
Sep 08 07:14:19.854540 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.854513    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-x2zfs"]
Sep 08 07:14:19.893859 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.893771    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-x2zfs"]
Sep 08 07:14:19.894044 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:14:19.894023    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cnpxp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-x2zfs" podUID="699bd875-fe5a-4987-8ee0-340099f83674"
Sep 08 07:14:19.924004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.923965    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-98b75"]
Sep 08 07:14:19.928745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.928709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:19.931331 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.931303    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 07:14:19.936615 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.936581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-98b75"]
Sep 08 07:14:19.980361 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:19.980324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpxp\" (UniqueName: \"kubernetes.io/projected/699bd875-fe5a-4987-8ee0-340099f83674-kube-api-access-cnpxp\") pod \"authorino-5dc7ccb5f-x2zfs\" (UID: \"699bd875-fe5a-4987-8ee0-340099f83674\") " pod="kuadrant-system/authorino-5dc7ccb5f-x2zfs"
Sep 08 07:14:20.073077 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.073032    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-98b75"]
Sep 08 07:14:20.073321 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:14:20.073302    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7fb2h tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-98b75" podUID="8ce7a787-1146-471f-9db3-6941d01258ba"
Sep 08 07:14:20.081716 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.081678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpxp\" (UniqueName: \"kubernetes.io/projected/699bd875-fe5a-4987-8ee0-340099f83674-kube-api-access-cnpxp\") pod \"authorino-5dc7ccb5f-x2zfs\" (UID: \"699bd875-fe5a-4987-8ee0-340099f83674\") " pod="kuadrant-system/authorino-5dc7ccb5f-x2zfs"
Sep 08 07:14:20.081844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.081821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fb2h\" (UniqueName: \"kubernetes.io/projected/8ce7a787-1146-471f-9db3-6941d01258ba-kube-api-access-7fb2h\") pod \"authorino-687bd45585-98b75\" (UID: \"8ce7a787-1146-471f-9db3-6941d01258ba\") " pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:20.081887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.081853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ce7a787-1146-471f-9db3-6941d01258ba-tls-cert\") pod \"authorino-687bd45585-98b75\" (UID: \"8ce7a787-1146-471f-9db3-6941d01258ba\") " pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:20.097682 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.097650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpxp\" (UniqueName: \"kubernetes.io/projected/699bd875-fe5a-4987-8ee0-340099f83674-kube-api-access-cnpxp\") pod \"authorino-5dc7ccb5f-x2zfs\" (UID: \"699bd875-fe5a-4987-8ee0-340099f83674\") " pod="kuadrant-system/authorino-5dc7ccb5f-x2zfs"
Sep 08 07:14:20.103073 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.103038    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8548c7d88f-m56n5"]
Sep 08 07:14:20.105679 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.105658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:14:20.115545 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.115518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8548c7d88f-m56n5"]
Sep 08 07:14:20.182819 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.182715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fb2h\" (UniqueName: \"kubernetes.io/projected/8ce7a787-1146-471f-9db3-6941d01258ba-kube-api-access-7fb2h\") pod \"authorino-687bd45585-98b75\" (UID: \"8ce7a787-1146-471f-9db3-6941d01258ba\") " pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:20.182819 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.182758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ce7a787-1146-471f-9db3-6941d01258ba-tls-cert\") pod \"authorino-687bd45585-98b75\" (UID: \"8ce7a787-1146-471f-9db3-6941d01258ba\") " pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:20.185358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.185328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ce7a787-1146-471f-9db3-6941d01258ba-tls-cert\") pod \"authorino-687bd45585-98b75\" (UID: \"8ce7a787-1146-471f-9db3-6941d01258ba\") " pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:20.190879 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.190855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fb2h\" (UniqueName: \"kubernetes.io/projected/8ce7a787-1146-471f-9db3-6941d01258ba-kube-api-access-7fb2h\") pod \"authorino-687bd45585-98b75\" (UID: \"8ce7a787-1146-471f-9db3-6941d01258ba\") " pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:20.283832 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.283796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgsx\" (UniqueName: \"kubernetes.io/projected/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-kube-api-access-ghgsx\") pod \"authorino-8548c7d88f-m56n5\" (UID: \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\") " pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:14:20.284069 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.283913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-tls-cert\") pod \"authorino-8548c7d88f-m56n5\" (UID: \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\") " pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:14:20.378035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.378004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:20.378035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.378019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-x2zfs"
Sep 08 07:14:20.384439 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.384414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:20.384626 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.384606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgsx\" (UniqueName: \"kubernetes.io/projected/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-kube-api-access-ghgsx\") pod \"authorino-8548c7d88f-m56n5\" (UID: \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\") " pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:14:20.384721 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.384707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-tls-cert\") pod \"authorino-8548c7d88f-m56n5\" (UID: \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\") " pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:14:20.387240 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.387212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-tls-cert\") pod \"authorino-8548c7d88f-m56n5\" (UID: \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\") " pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:14:20.388033 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.388011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-x2zfs"
Sep 08 07:14:20.395813 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.395782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgsx\" (UniqueName: \"kubernetes.io/projected/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-kube-api-access-ghgsx\") pod \"authorino-8548c7d88f-m56n5\" (UID: \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\") " pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:14:20.414557 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.414519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:14:20.486345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.485941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fb2h\" (UniqueName: \"kubernetes.io/projected/8ce7a787-1146-471f-9db3-6941d01258ba-kube-api-access-7fb2h\") pod \"8ce7a787-1146-471f-9db3-6941d01258ba\" (UID: \"8ce7a787-1146-471f-9db3-6941d01258ba\") "
Sep 08 07:14:20.486345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.486041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ce7a787-1146-471f-9db3-6941d01258ba-tls-cert\") pod \"8ce7a787-1146-471f-9db3-6941d01258ba\" (UID: \"8ce7a787-1146-471f-9db3-6941d01258ba\") "
Sep 08 07:14:20.486345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.486108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnpxp\" (UniqueName: \"kubernetes.io/projected/699bd875-fe5a-4987-8ee0-340099f83674-kube-api-access-cnpxp\") pod \"699bd875-fe5a-4987-8ee0-340099f83674\" (UID: \"699bd875-fe5a-4987-8ee0-340099f83674\") "
Sep 08 07:14:20.488219 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.488174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce7a787-1146-471f-9db3-6941d01258ba-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8ce7a787-1146-471f-9db3-6941d01258ba" (UID: "8ce7a787-1146-471f-9db3-6941d01258ba"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:14:20.488344 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.488216    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce7a787-1146-471f-9db3-6941d01258ba-kube-api-access-7fb2h" (OuterVolumeSpecName: "kube-api-access-7fb2h") pod "8ce7a787-1146-471f-9db3-6941d01258ba" (UID: "8ce7a787-1146-471f-9db3-6941d01258ba"). InnerVolumeSpecName "kube-api-access-7fb2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:14:20.488344 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.488255    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699bd875-fe5a-4987-8ee0-340099f83674-kube-api-access-cnpxp" (OuterVolumeSpecName: "kube-api-access-cnpxp") pod "699bd875-fe5a-4987-8ee0-340099f83674" (UID: "699bd875-fe5a-4987-8ee0-340099f83674"). InnerVolumeSpecName "kube-api-access-cnpxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:14:20.587784 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.587745    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ce7a787-1146-471f-9db3-6941d01258ba-tls-cert\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:14:20.587784 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.587776    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnpxp\" (UniqueName: \"kubernetes.io/projected/699bd875-fe5a-4987-8ee0-340099f83674-kube-api-access-cnpxp\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:14:20.587784 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.587787    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fb2h\" (UniqueName: \"kubernetes.io/projected/8ce7a787-1146-471f-9db3-6941d01258ba-kube-api-access-7fb2h\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:14:20.683248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:20.683208    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8548c7d88f-m56n5"]
Sep 08 07:14:20.686943 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:14:20.686914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae1043e9_7a48_4b4d_8ed7_476c065cbb65.slice/crio-c9f5abb2b537b3fc4940e9b607a6ee8d24c803a95540424301baa816f02a8034 WatchSource:0}: Error finding container c9f5abb2b537b3fc4940e9b607a6ee8d24c803a95540424301baa816f02a8034: Status 404 returned error can't find the container with id c9f5abb2b537b3fc4940e9b607a6ee8d24c803a95540424301baa816f02a8034
Sep 08 07:14:21.384084 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.384052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-x2zfs"
Sep 08 07:14:21.384084 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.384072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8548c7d88f-m56n5" event={"ID":"ae1043e9-7a48-4b4d-8ed7-476c065cbb65","Type":"ContainerStarted","Data":"c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932"}
Sep 08 07:14:21.384337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.384113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8548c7d88f-m56n5" event={"ID":"ae1043e9-7a48-4b4d-8ed7-476c065cbb65","Type":"ContainerStarted","Data":"c9f5abb2b537b3fc4940e9b607a6ee8d24c803a95540424301baa816f02a8034"}
Sep 08 07:14:21.384337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.384212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-98b75"
Sep 08 07:14:21.412676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.412605    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8548c7d88f-m56n5" podStartSLOduration=0.790708283 podStartE2EDuration="1.412578555s" totalImagesPullingTime="621.870272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:14:21.400363916 +0000 UTC m=+577.083824073" watchObservedRunningTime="2026-09-08 07:14:21.412578555 +0000 UTC m=+577.096038710"
Sep 08 07:14:21.424035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.423957    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-x2zfs"]
Sep 08 07:14:21.427096 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.427063    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-v42mn"]
Sep 08 07:14:21.427338 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.427282    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-v42mn" podUID="2ea560ec-12ce-4998-8d8d-6ff3e65835dc" containerName="authorino" containerID="cri-o://7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e" gracePeriod=30
Sep 08 07:14:21.429687 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.429665    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-x2zfs"]
Sep 08 07:14:21.459201 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.459168    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-98b75"]
Sep 08 07:14:21.461211 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.461176    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-98b75"]
Sep 08 07:14:21.660897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.660870    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-v42mn"
Sep 08 07:14:21.797679 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.797638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb52b\" (UniqueName: \"kubernetes.io/projected/2ea560ec-12ce-4998-8d8d-6ff3e65835dc-kube-api-access-tb52b\") pod \"2ea560ec-12ce-4998-8d8d-6ff3e65835dc\" (UID: \"2ea560ec-12ce-4998-8d8d-6ff3e65835dc\") "
Sep 08 07:14:21.799709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.799686    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea560ec-12ce-4998-8d8d-6ff3e65835dc-kube-api-access-tb52b" (OuterVolumeSpecName: "kube-api-access-tb52b") pod "2ea560ec-12ce-4998-8d8d-6ff3e65835dc" (UID: "2ea560ec-12ce-4998-8d8d-6ff3e65835dc"). InnerVolumeSpecName "kube-api-access-tb52b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:14:21.899325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:21.899230    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb52b\" (UniqueName: \"kubernetes.io/projected/2ea560ec-12ce-4998-8d8d-6ff3e65835dc-kube-api-access-tb52b\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:14:22.392606 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.392564    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ea560ec-12ce-4998-8d8d-6ff3e65835dc" containerID="7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e" exitCode=0
Sep 08 07:14:22.392606 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.392606    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-v42mn"
Sep 08 07:14:22.392883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.392607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-v42mn" event={"ID":"2ea560ec-12ce-4998-8d8d-6ff3e65835dc","Type":"ContainerDied","Data":"7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e"}
Sep 08 07:14:22.392883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.392659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-v42mn" event={"ID":"2ea560ec-12ce-4998-8d8d-6ff3e65835dc","Type":"ContainerDied","Data":"7a1284a915f071334bf91e482f71032561ec2725834e36f47343c93ee53bf857"}
Sep 08 07:14:22.392883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.392683    2580 scope.go:117] "RemoveContainer" containerID="7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e"
Sep 08 07:14:22.401814 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.401793    2580 scope.go:117] "RemoveContainer" containerID="7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e"
Sep 08 07:14:22.402140 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:14:22.402121    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e\": container with ID starting with 7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e not found: ID does not exist" containerID="7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e"
Sep 08 07:14:22.402212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.402157    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e"} err="failed to get container status \"7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e\": rpc error: code = NotFound desc = could not find container \"7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e\": container with ID starting with 7817c165999e621f6a06e8299b608d5018961deff2d7153d30c72e10f1ecd84e not found: ID does not exist"
Sep 08 07:14:22.412368 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.412335    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-v42mn"]
Sep 08 07:14:22.418303 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.418273    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-v42mn"]
Sep 08 07:14:22.940728 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.940693    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea560ec-12ce-4998-8d8d-6ff3e65835dc" path="/var/lib/kubelet/pods/2ea560ec-12ce-4998-8d8d-6ff3e65835dc/volumes"
Sep 08 07:14:22.941108 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.941008    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699bd875-fe5a-4987-8ee0-340099f83674" path="/var/lib/kubelet/pods/699bd875-fe5a-4987-8ee0-340099f83674/volumes"
Sep 08 07:14:22.941240 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:22.941227    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce7a787-1146-471f-9db3-6941d01258ba" path="/var/lib/kubelet/pods/8ce7a787-1146-471f-9db3-6941d01258ba/volumes"
Sep 08 07:14:25.861591 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:25.861557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:27.045723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:27.045690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:29.193265 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:29.193236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:29.291780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:29.291743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:59.695142 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:59.695105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:14:59.795663 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:14:59.795624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:00.779493 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:00.779447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:00.877961 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:00.877927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:02.324864 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:02.324824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:02.445452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:02.445412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:02.545789 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:02.545750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:02.649802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:02.649695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:02.845084 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:02.845051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:02.952845 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:02.952757    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:12.552876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:12.552798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:12.681875 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:12.681843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:17.745635 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:17.745602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:25.845930 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.845898    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"]
Sep 08 07:15:25.846347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.846211    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea560ec-12ce-4998-8d8d-6ff3e65835dc" containerName="authorino"
Sep 08 07:15:25.846347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.846223    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea560ec-12ce-4998-8d8d-6ff3e65835dc" containerName="authorino"
Sep 08 07:15:25.846347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.846285    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea560ec-12ce-4998-8d8d-6ff3e65835dc" containerName="authorino"
Sep 08 07:15:25.848684 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.848666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:25.851018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.850998    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 07:15:25.851018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.851009    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 07:15:25.851647 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.851628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zg7w8\""
Sep 08 07:15:25.851774 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.851657    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 07:15:25.858973 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.858951    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"]
Sep 08 07:15:25.884378 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.884351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:25.944159 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.944122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e01955d1-bdf8-41c7-a042-e1bbeb7af050-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:25.944326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.944171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz4nw\" (UniqueName: \"kubernetes.io/projected/e01955d1-bdf8-41c7-a042-e1bbeb7af050-kube-api-access-xz4nw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:25.944326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.944241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:25.944326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.944276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:25.944326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.944317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:25.944504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:25.944342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.045384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.045343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.045384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.045391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.045654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.045491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e01955d1-bdf8-41c7-a042-e1bbeb7af050-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.045654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.045510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz4nw\" (UniqueName: \"kubernetes.io/projected/e01955d1-bdf8-41c7-a042-e1bbeb7af050-kube-api-access-xz4nw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.045654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.045555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.045654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.045605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.045868 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.045786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.046052 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.046028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.046154 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.046071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.047647 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.047622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e01955d1-bdf8-41c7-a042-e1bbeb7af050-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.048110 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.048093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e01955d1-bdf8-41c7-a042-e1bbeb7af050-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.054047 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.054025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz4nw\" (UniqueName: \"kubernetes.io/projected/e01955d1-bdf8-41c7-a042-e1bbeb7af050-kube-api-access-xz4nw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9\" (UID: \"e01955d1-bdf8-41c7-a042-e1bbeb7af050\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.159341 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.159263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:26.428285 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.428261    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"]
Sep 08 07:15:26.431486 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:15:26.431431    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01955d1_bdf8_41c7_a042_e1bbeb7af050.slice/crio-611bdfb59bcb76000066db09f96fe4f8b0dac638c31031a545d7841a4f07f95c WatchSource:0}: Error finding container 611bdfb59bcb76000066db09f96fe4f8b0dac638c31031a545d7841a4f07f95c: Status 404 returned error can't find the container with id 611bdfb59bcb76000066db09f96fe4f8b0dac638c31031a545d7841a4f07f95c
Sep 08 07:15:26.433081 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.433062    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 07:15:26.669796 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:26.669757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9" event={"ID":"e01955d1-bdf8-41c7-a042-e1bbeb7af050","Type":"ContainerStarted","Data":"611bdfb59bcb76000066db09f96fe4f8b0dac638c31031a545d7841a4f07f95c"}
Sep 08 07:15:27.676229 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:27.676186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9" event={"ID":"e01955d1-bdf8-41c7-a042-e1bbeb7af050","Type":"ContainerStarted","Data":"15cddb7a7561d4a6f64ac35f572f0be639c88d00b026148caad3974756b02de2"}
Sep 08 07:15:27.693626 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:27.693559    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9" podStartSLOduration=1.543071906 podStartE2EDuration="2.693539226s" totalImagesPullingTime="1.15046732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:15:27.691919319 +0000 UTC m=+643.375379472" watchObservedRunningTime="2026-09-08 07:15:27.693539226 +0000 UTC m=+643.376999380"
Sep 08 07:15:35.361135 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:35.361104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:36.159618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:36.159583    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:36.159804 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:36.159649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:36.173071 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:36.173044    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:36.725812 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:36.725777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9"
Sep 08 07:15:55.045105 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:55.045073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:15:55.678755 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:15:55.678719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:16:09.244309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.244276    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"]
Sep 08 07:16:09.247657 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.247639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.249695 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.249674    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 08 07:16:09.259387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.259363    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"]
Sep 08 07:16:09.285398 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.285366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:16:09.325195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.325161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.325195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.325197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.325358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.325216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829d1e3c-c13d-466a-8a04-175993c07f8d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.325358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.325322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzcd9\" (UniqueName: \"kubernetes.io/projected/829d1e3c-c13d-466a-8a04-175993c07f8d-kube-api-access-vzcd9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.325430 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.325360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.325430 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.325406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.426620 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.426580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzcd9\" (UniqueName: \"kubernetes.io/projected/829d1e3c-c13d-466a-8a04-175993c07f8d-kube-api-access-vzcd9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.426763 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.426630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.426860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.426837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.426951 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.426938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.427011 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.426970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.427011 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.426996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829d1e3c-c13d-466a-8a04-175993c07f8d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.427011 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.427003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.427204 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.427185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.427317 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.427287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.429048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.429030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829d1e3c-c13d-466a-8a04-175993c07f8d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.429322 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.429305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829d1e3c-c13d-466a-8a04-175993c07f8d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.434767 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.434738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzcd9\" (UniqueName: \"kubernetes.io/projected/829d1e3c-c13d-466a-8a04-175993c07f8d-kube-api-access-vzcd9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c97v4\" (UID: \"829d1e3c-c13d-466a-8a04-175993c07f8d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.560238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.560144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:09.828638 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.828560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"]
Sep 08 07:16:09.830899 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:16:09.830872    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829d1e3c_c13d_466a_8a04_175993c07f8d.slice/crio-f3327cbb6c88cac808139181aee2f3f6bc857ea2ea020d64843915d538cf41fa WatchSource:0}: Error finding container f3327cbb6c88cac808139181aee2f3f6bc857ea2ea020d64843915d538cf41fa: Status 404 returned error can't find the container with id f3327cbb6c88cac808139181aee2f3f6bc857ea2ea020d64843915d538cf41fa
Sep 08 07:16:09.857930 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:09.857900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4" event={"ID":"829d1e3c-c13d-466a-8a04-175993c07f8d","Type":"ContainerStarted","Data":"f3327cbb6c88cac808139181aee2f3f6bc857ea2ea020d64843915d538cf41fa"}
Sep 08 07:16:10.863796 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:10.863758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4" event={"ID":"829d1e3c-c13d-466a-8a04-175993c07f8d","Type":"ContainerStarted","Data":"4adea2ec5faa23a25f5aa323ca3497adf25b0f53cf5ec2649ad50d7357523d1e"}
Sep 08 07:16:10.883178 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:10.883107    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4" podStartSLOduration=1.684562071 podStartE2EDuration="1.883085519s" totalImagesPullingTime="198.523448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:16:10.881258908 +0000 UTC m=+686.564719066" watchObservedRunningTime="2026-09-08 07:16:10.883085519 +0000 UTC m=+686.566545673"
Sep 08 07:16:19.168824 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:19.168773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:16:19.561237 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:19.561153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:19.561237 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:19.561229    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:19.573842 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:19.573808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:19.915949 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:19.915915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c97v4"
Sep 08 07:16:37.440769 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.440692    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"]
Sep 08 07:16:37.443358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.443332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.445360 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.445341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 07:16:37.456352 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.456327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"]
Sep 08 07:16:37.468948 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.468925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:16:37.573134 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.573078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82j8\" (UniqueName: \"kubernetes.io/projected/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-kube-api-access-l82j8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.573134 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.573138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.573387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.573165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.573387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.573219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.573387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.573334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.573387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.573371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.674866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.674812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.674866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.674872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.675127 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.674927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82j8\" (UniqueName: \"kubernetes.io/projected/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-kube-api-access-l82j8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.675127 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.674957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.675127 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.674979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.675127 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.675011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.675432 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.675408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.675525 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.675494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.675525 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.675500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.677418 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.677393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.677507 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.677439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.682501 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.682479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82j8\" (UniqueName: \"kubernetes.io/projected/c61fcf99-aff9-4df8-9fd3-28c34619a7e5-kube-api-access-l82j8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7ms5d\" (UID: \"c61fcf99-aff9-4df8-9fd3-28c34619a7e5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:37.754217 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:37.754123    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:38.025703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:38.025669    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"]
Sep 08 07:16:38.027824 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:16:38.027799    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61fcf99_aff9_4df8_9fd3_28c34619a7e5.slice/crio-af716cee74e73554f9cd275db1f5ae16f0aecbc3a713c879d3bc0aee7c60795a WatchSource:0}: Error finding container af716cee74e73554f9cd275db1f5ae16f0aecbc3a713c879d3bc0aee7c60795a: Status 404 returned error can't find the container with id af716cee74e73554f9cd275db1f5ae16f0aecbc3a713c879d3bc0aee7c60795a
Sep 08 07:16:38.989915 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:38.989877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d" event={"ID":"c61fcf99-aff9-4df8-9fd3-28c34619a7e5","Type":"ContainerStarted","Data":"3b5785283845ee42b1de645d1ab45dc7c78e3508d312186231274b4a82230187"}
Sep 08 07:16:38.990304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:38.989940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d" event={"ID":"c61fcf99-aff9-4df8-9fd3-28c34619a7e5","Type":"ContainerStarted","Data":"af716cee74e73554f9cd275db1f5ae16f0aecbc3a713c879d3bc0aee7c60795a"}
Sep 08 07:16:39.007156 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:39.007110    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d" podStartSLOduration=1.8302257069999999 podStartE2EDuration="2.007093917s" totalImagesPullingTime="176.86821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:16:39.006084824 +0000 UTC m=+714.689544973" watchObservedRunningTime="2026-09-08 07:16:39.007093917 +0000 UTC m=+714.690554070"
Sep 08 07:16:40.906452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:40.906418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:16:40.956399 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:40.956371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:16:44.546853 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.546819    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"]
Sep 08 07:16:44.550268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.550248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.552249 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.552225    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 07:16:44.559186 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.559160    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"]
Sep 08 07:16:44.575478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.575432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:16:44.641053 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.641017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.641208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.641062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb705ec-c8b9-49f7-95b7-dac2f6002e26-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.641208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.641107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.641208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.641128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.641208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.641143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nq2\" (UniqueName: \"kubernetes.io/projected/efb705ec-c8b9-49f7-95b7-dac2f6002e26-kube-api-access-89nq2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.641208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.641199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.741670 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.741632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.741844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.741678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb705ec-c8b9-49f7-95b7-dac2f6002e26-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.741844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.741715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.741844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.741741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.741844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.741764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nq2\" (UniqueName: \"kubernetes.io/projected/efb705ec-c8b9-49f7-95b7-dac2f6002e26-kube-api-access-89nq2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.741844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.741804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.742097 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.741983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.742097 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.742048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.742201 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.742164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.743919 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.743889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb705ec-c8b9-49f7-95b7-dac2f6002e26-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.744160 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.744144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 07:16:44.750018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.749993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nq2\" (UniqueName: \"kubernetes.io/projected/efb705ec-c8b9-49f7-95b7-dac2f6002e26-kube-api-access-89nq2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.754577 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.754556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb705ec-c8b9-49f7-95b7-dac2f6002e26-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-75ggk\" (UID: \"efb705ec-c8b9-49f7-95b7-dac2f6002e26\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:44.860158 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:44.860124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:45.132777 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:45.132753    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"]
Sep 08 07:16:45.135367 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:16:45.135341    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefb705ec_c8b9_49f7_95b7_dac2f6002e26.slice/crio-884de561ce5e882aa0251b3132ae5613396a7bc024211e2852d299f335779efe WatchSource:0}: Error finding container 884de561ce5e882aa0251b3132ae5613396a7bc024211e2852d299f335779efe: Status 404 returned error can't find the container with id 884de561ce5e882aa0251b3132ae5613396a7bc024211e2852d299f335779efe
Sep 08 07:16:46.023290 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:46.023257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk" event={"ID":"efb705ec-c8b9-49f7-95b7-dac2f6002e26","Type":"ContainerStarted","Data":"c964e3045350b504aebb54de4a3fdc9dc90f374e605c99bf3593ed07aee7c135"}
Sep 08 07:16:46.023684 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:46.023300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk" event={"ID":"efb705ec-c8b9-49f7-95b7-dac2f6002e26","Type":"ContainerStarted","Data":"884de561ce5e882aa0251b3132ae5613396a7bc024211e2852d299f335779efe"}
Sep 08 07:16:46.043359 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:46.043301    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk" podStartSLOduration=1.867968511 podStartE2EDuration="2.04328353s" totalImagesPullingTime="175.315019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:16:46.041921917 +0000 UTC m=+721.725382081" watchObservedRunningTime="2026-09-08 07:16:46.04328353 +0000 UTC m=+721.726743683"
Sep 08 07:16:47.754946 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:47.754909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:47.755743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:47.755720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:47.773565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:47.773535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:48.042606 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:48.042525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7ms5d"
Sep 08 07:16:54.860959 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:54.860926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:54.860959 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:54.860970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:54.873484 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:54.873438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:16:55.072092 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:16:55.072066    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-75ggk"
Sep 08 07:17:57.346362 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:17:57.346326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:00.330023 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.329988    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-648b45f788-rjlcc"]
Sep 08 07:18:00.333228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.333211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648b45f788-rjlcc"
Sep 08 07:18:00.341728 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.341699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-648b45f788-rjlcc"]
Sep 08 07:18:00.398199 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.398161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54558\" (UniqueName: \"kubernetes.io/projected/87260785-3347-4707-b458-fc07bbcc8104-kube-api-access-54558\") pod \"authorino-648b45f788-rjlcc\" (UID: \"87260785-3347-4707-b458-fc07bbcc8104\") " pod="kuadrant-system/authorino-648b45f788-rjlcc"
Sep 08 07:18:00.398364 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.398242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87260785-3347-4707-b458-fc07bbcc8104-tls-cert\") pod \"authorino-648b45f788-rjlcc\" (UID: \"87260785-3347-4707-b458-fc07bbcc8104\") " pod="kuadrant-system/authorino-648b45f788-rjlcc"
Sep 08 07:18:00.499527 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.499454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87260785-3347-4707-b458-fc07bbcc8104-tls-cert\") pod \"authorino-648b45f788-rjlcc\" (UID: \"87260785-3347-4707-b458-fc07bbcc8104\") " pod="kuadrant-system/authorino-648b45f788-rjlcc"
Sep 08 07:18:00.499688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.499583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54558\" (UniqueName: \"kubernetes.io/projected/87260785-3347-4707-b458-fc07bbcc8104-kube-api-access-54558\") pod \"authorino-648b45f788-rjlcc\" (UID: \"87260785-3347-4707-b458-fc07bbcc8104\") " pod="kuadrant-system/authorino-648b45f788-rjlcc"
Sep 08 07:18:00.501856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.501830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87260785-3347-4707-b458-fc07bbcc8104-tls-cert\") pod \"authorino-648b45f788-rjlcc\" (UID: \"87260785-3347-4707-b458-fc07bbcc8104\") " pod="kuadrant-system/authorino-648b45f788-rjlcc"
Sep 08 07:18:00.507902 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.507870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54558\" (UniqueName: \"kubernetes.io/projected/87260785-3347-4707-b458-fc07bbcc8104-kube-api-access-54558\") pod \"authorino-648b45f788-rjlcc\" (UID: \"87260785-3347-4707-b458-fc07bbcc8104\") " pod="kuadrant-system/authorino-648b45f788-rjlcc"
Sep 08 07:18:00.642740 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.642656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648b45f788-rjlcc"
Sep 08 07:18:00.914697 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:00.914672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-648b45f788-rjlcc"]
Sep 08 07:18:00.917301 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:18:00.917274    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87260785_3347_4707_b458_fc07bbcc8104.slice/crio-1312ee7727c91821c35b3fb8567c6dd2370e61cdc8ba351f69837de23c0a8d75 WatchSource:0}: Error finding container 1312ee7727c91821c35b3fb8567c6dd2370e61cdc8ba351f69837de23c0a8d75: Status 404 returned error can't find the container with id 1312ee7727c91821c35b3fb8567c6dd2370e61cdc8ba351f69837de23c0a8d75
Sep 08 07:18:01.352853 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:01.352816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648b45f788-rjlcc" event={"ID":"87260785-3347-4707-b458-fc07bbcc8104","Type":"ContainerStarted","Data":"1312ee7727c91821c35b3fb8567c6dd2370e61cdc8ba351f69837de23c0a8d75"}
Sep 08 07:18:02.362516 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.362479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648b45f788-rjlcc" event={"ID":"87260785-3347-4707-b458-fc07bbcc8104","Type":"ContainerStarted","Data":"8f9fa80db9515b5269a29859b267ef8f434f0e8e78c640aaa94eae4b929db183"}
Sep 08 07:18:02.377228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.377173    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-648b45f788-rjlcc" podStartSLOduration=1.410552609 podStartE2EDuration="2.377150708s" totalImagesPullingTime="966.598099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:18:02.376569387 +0000 UTC m=+798.060029542" watchObservedRunningTime="2026-09-08 07:18:02.377150708 +0000 UTC m=+798.060610862"
Sep 08 07:18:02.408286 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.408243    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8548c7d88f-m56n5"]
Sep 08 07:18:02.408817 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.408451    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8548c7d88f-m56n5" podUID="ae1043e9-7a48-4b4d-8ed7-476c065cbb65" containerName="authorino" containerID="cri-o://c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932" gracePeriod=30
Sep 08 07:18:02.656122 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.656092    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:18:02.725835 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.725801    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-tls-cert\") pod \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\" (UID: \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\") "
Sep 08 07:18:02.726003 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.725942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghgsx\" (UniqueName: \"kubernetes.io/projected/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-kube-api-access-ghgsx\") pod \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\" (UID: \"ae1043e9-7a48-4b4d-8ed7-476c065cbb65\") "
Sep 08 07:18:02.727930 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.727894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-kube-api-access-ghgsx" (OuterVolumeSpecName: "kube-api-access-ghgsx") pod "ae1043e9-7a48-4b4d-8ed7-476c065cbb65" (UID: "ae1043e9-7a48-4b4d-8ed7-476c065cbb65"). InnerVolumeSpecName "kube-api-access-ghgsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:18:02.735570 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.735538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ae1043e9-7a48-4b4d-8ed7-476c065cbb65" (UID: "ae1043e9-7a48-4b4d-8ed7-476c065cbb65"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:18:02.827718 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.827670    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-tls-cert\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:18:02.827718 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.827717    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghgsx\" (UniqueName: \"kubernetes.io/projected/ae1043e9-7a48-4b4d-8ed7-476c065cbb65-kube-api-access-ghgsx\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:18:02.879736 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.879703    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:02.979957 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:02.979923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:03.368770 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.368732    2580 generic.go:356] "Generic (PLEG): container finished" podID="ae1043e9-7a48-4b4d-8ed7-476c065cbb65" containerID="c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932" exitCode=0
Sep 08 07:18:03.369207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.368789    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8548c7d88f-m56n5"
Sep 08 07:18:03.369207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.368816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8548c7d88f-m56n5" event={"ID":"ae1043e9-7a48-4b4d-8ed7-476c065cbb65","Type":"ContainerDied","Data":"c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932"}
Sep 08 07:18:03.369207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.368858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8548c7d88f-m56n5" event={"ID":"ae1043e9-7a48-4b4d-8ed7-476c065cbb65","Type":"ContainerDied","Data":"c9f5abb2b537b3fc4940e9b607a6ee8d24c803a95540424301baa816f02a8034"}
Sep 08 07:18:03.369207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.368875    2580 scope.go:117] "RemoveContainer" containerID="c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932"
Sep 08 07:18:03.377392 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.377375    2580 scope.go:117] "RemoveContainer" containerID="c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932"
Sep 08 07:18:03.377732 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:18:03.377712    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932\": container with ID starting with c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932 not found: ID does not exist" containerID="c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932"
Sep 08 07:18:03.377786 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.377742    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932"} err="failed to get container status \"c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932\": rpc error: code = NotFound desc = could not find container \"c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932\": container with ID starting with c3409290c641eafb8dcbc6eafcb9723ff1b4e144bcc36d2e7356643d0bda5932 not found: ID does not exist"
Sep 08 07:18:03.386588 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.386564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8548c7d88f-m56n5"]
Sep 08 07:18:03.390142 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.390120    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8548c7d88f-m56n5"]
Sep 08 07:18:03.988345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:03.988311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:04.087990 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:04.087958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:04.942257 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:04.942225    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae1043e9-7a48-4b4d-8ed7-476c065cbb65" path="/var/lib/kubelet/pods/ae1043e9-7a48-4b4d-8ed7-476c065cbb65/volumes"
Sep 08 07:18:19.235780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:19.235702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:19.343615 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:19.343579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:19.543417 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:19.543345    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:19.736277 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:19.736241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:20.553633 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:20.553602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:20.658469 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:20.658424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:21.000510 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:21.000478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:22.735379 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:22.735344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:22.834507 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:22.834480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:23.893693 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:23.893661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:23.949652 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:23.949615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:24.336405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:24.336374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:24.438703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:24.438669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:24.538126 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:24.538093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:26.933914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:26.933878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:27.036275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:27.036241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:27.149582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:27.149534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:27.261015 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:27.260935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:27.445521 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:27.445492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:27.618596 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:27.618561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:28.020309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:28.020224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:28.121986 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:28.121953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:30.254947 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:30.254908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:30.346215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:30.346180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:31.428730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:31.428678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:31.530624 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:31.530587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:31.665418 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:31.665382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:32.017826 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.017792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:32.117588 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.117550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:32.157670 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.157637    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"]
Sep 08 07:18:32.157987 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.157974    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1043e9-7a48-4b4d-8ed7-476c065cbb65" containerName="authorino"
Sep 08 07:18:32.158043 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.157988    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1043e9-7a48-4b4d-8ed7-476c065cbb65" containerName="authorino"
Sep 08 07:18:32.158080 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.158045    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1043e9-7a48-4b4d-8ed7-476c065cbb65" containerName="authorino"
Sep 08 07:18:32.160819 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.160802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:32.162845 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.162827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4a1fb56c-dockercfg-sh4fj\""
Sep 08 07:18:32.162914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.162851    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4a1fb56c\""
Sep 08 07:18:32.170084 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.170059    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"]
Sep 08 07:18:32.190419 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.190390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbq4m\" (UniqueName: \"kubernetes.io/projected/998180da-833e-4981-9684-e9ea30c2ad44-kube-api-access-jbq4m\") pod \"payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl\" (UID: \"998180da-833e-4981-9684-e9ea30c2ad44\") " pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:32.190587 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.190436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998180da-833e-4981-9684-e9ea30c2ad44-plugins-config-volume\") pod \"payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl\" (UID: \"998180da-833e-4981-9684-e9ea30c2ad44\") " pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:32.291863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.291765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbq4m\" (UniqueName: \"kubernetes.io/projected/998180da-833e-4981-9684-e9ea30c2ad44-kube-api-access-jbq4m\") pod \"payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl\" (UID: \"998180da-833e-4981-9684-e9ea30c2ad44\") " pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:32.291863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.291817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998180da-833e-4981-9684-e9ea30c2ad44-plugins-config-volume\") pod \"payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl\" (UID: \"998180da-833e-4981-9684-e9ea30c2ad44\") " pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:32.292504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.292486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998180da-833e-4981-9684-e9ea30c2ad44-plugins-config-volume\") pod \"payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl\" (UID: \"998180da-833e-4981-9684-e9ea30c2ad44\") " pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:32.299634 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.299611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbq4m\" (UniqueName: \"kubernetes.io/projected/998180da-833e-4981-9684-e9ea30c2ad44-kube-api-access-jbq4m\") pod \"payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl\" (UID: \"998180da-833e-4981-9684-e9ea30c2ad44\") " pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:32.472317 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.472282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:32.745321 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:32.745286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"]
Sep 08 07:18:32.748197 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:18:32.748167    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998180da_833e_4981_9684_e9ea30c2ad44.slice/crio-07e8228a9b395fcaade53fb2e95c03731eb995b1788ccc66d7b74389bd5b0933 WatchSource:0}: Error finding container 07e8228a9b395fcaade53fb2e95c03731eb995b1788ccc66d7b74389bd5b0933: Status 404 returned error can't find the container with id 07e8228a9b395fcaade53fb2e95c03731eb995b1788ccc66d7b74389bd5b0933
Sep 08 07:18:33.518455 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:33.518416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl" event={"ID":"998180da-833e-4981-9684-e9ea30c2ad44","Type":"ContainerStarted","Data":"07e8228a9b395fcaade53fb2e95c03731eb995b1788ccc66d7b74389bd5b0933"}
Sep 08 07:18:33.519844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:33.519815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:33.803498 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:33.803388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:34.679838 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:34.679788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:34.746392 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:34.746346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:35.020471 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.020416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:35.181108 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.181075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:35.280690 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.280603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:35.324486 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.324431    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"]
Sep 08 07:18:35.328032 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.328000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:35.329981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.329952    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-638d2d-dockercfg-nz7ps\""
Sep 08 07:18:35.329981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.329959    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-638d2d\""
Sep 08 07:18:35.339033 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.339007    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"]
Sep 08 07:18:35.424093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.424056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befb5eb-9437-41c3-bb59-e5b3884c3df4-plugins-config-volume\") pod \"payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq\" (UID: \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\") " pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:35.424297 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.424191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7k5n\" (UniqueName: \"kubernetes.io/projected/3befb5eb-9437-41c3-bb59-e5b3884c3df4-kube-api-access-b7k5n\") pod \"payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq\" (UID: \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\") " pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:35.524953 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.524919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befb5eb-9437-41c3-bb59-e5b3884c3df4-plugins-config-volume\") pod \"payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq\" (UID: \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\") " pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:35.525150 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.525004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7k5n\" (UniqueName: \"kubernetes.io/projected/3befb5eb-9437-41c3-bb59-e5b3884c3df4-kube-api-access-b7k5n\") pod \"payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq\" (UID: \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\") " pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:35.525765 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.525744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befb5eb-9437-41c3-bb59-e5b3884c3df4-plugins-config-volume\") pod \"payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq\" (UID: \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\") " pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:35.530275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.530247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl" event={"ID":"998180da-833e-4981-9684-e9ea30c2ad44","Type":"ContainerStarted","Data":"707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e"}
Sep 08 07:18:35.530429 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.530364    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:35.533376 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.533322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7k5n\" (UniqueName: \"kubernetes.io/projected/3befb5eb-9437-41c3-bb59-e5b3884c3df4-kube-api-access-b7k5n\") pod \"payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq\" (UID: \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\") " pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:35.547284 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.547228    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl" podStartSLOduration=1.291475768 podStartE2EDuration="3.547210802s" totalImagesPullingTime="2.255735034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:18:35.545144381 +0000 UTC m=+831.228604530" watchObservedRunningTime="2026-09-08 07:18:35.547210802 +0000 UTC m=+831.230670954"
Sep 08 07:18:35.638871 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.638836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:35.912890 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:35.912825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"]
Sep 08 07:18:35.916052 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:18:35.916023    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3befb5eb_9437_41c3_bb59_e5b3884c3df4.slice/crio-b0eceb56c58586eb65011b75f1592dfe7d7b04dcde6b71f83389a8a3a0754407 WatchSource:0}: Error finding container b0eceb56c58586eb65011b75f1592dfe7d7b04dcde6b71f83389a8a3a0754407: Status 404 returned error can't find the container with id b0eceb56c58586eb65011b75f1592dfe7d7b04dcde6b71f83389a8a3a0754407
Sep 08 07:18:36.537264 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:36.537225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq" event={"ID":"3befb5eb-9437-41c3-bb59-e5b3884c3df4","Type":"ContainerStarted","Data":"9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925"}
Sep 08 07:18:36.537519 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:36.537291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq" event={"ID":"3befb5eb-9437-41c3-bb59-e5b3884c3df4","Type":"ContainerStarted","Data":"b0eceb56c58586eb65011b75f1592dfe7d7b04dcde6b71f83389a8a3a0754407"}
Sep 08 07:18:36.537615 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:36.537596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:36.553720 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:36.553660    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq" podStartSLOduration=1.553641853 podStartE2EDuration="1.553641853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:18:36.552862137 +0000 UTC m=+832.236322300" watchObservedRunningTime="2026-09-08 07:18:36.553641853 +0000 UTC m=+832.237102025"
Sep 08 07:18:37.886429 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:37.886389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:38.037423 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:38.037388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:38.136974 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:38.136893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:38.331651 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:38.331594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:40.850212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:40.850154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:42.377767 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:42.377722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:44.100725 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:44.100687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:44.210561 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:44.210525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:44.316763 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:44.316717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:44.855885 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:44.855836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:46.539630 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:46.539589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:18:47.543563 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:47.543529    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:18:47.598566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:47.598530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:47.903596 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:47.903567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:48.281955 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:48.281838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:48.380540 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:48.380502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:50.923775 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:50.923740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:51.015347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:51.015319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:51.453445 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:51.453414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:51.560893 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:51.560864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:51.650087 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:51.650055    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:51.763764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:51.763674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:52.322676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.322641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:52.422727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.422692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:52.684005 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.683909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:52.779856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.779812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:52.808841 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.808816    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"]
Sep 08 07:18:52.813242 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.813208    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:52.815147 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.815130    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6f5c4f\""
Sep 08 07:18:52.823423 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.823403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"]
Sep 08 07:18:52.883440 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.883405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqd9\" (UniqueName: \"kubernetes.io/projected/b1e84e93-1464-4401-b3d4-a3d2da78bfca-kube-api-access-vkqd9\") pod \"payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc\" (UID: \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:52.883617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.883549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1e84e93-1464-4401-b3d4-a3d2da78bfca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc\" (UID: \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:52.892027 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.892002    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"]
Sep 08 07:18:52.895280 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.895266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:52.897170 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.897150    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6f5c4f-dockercfg-zpgpr\""
Sep 08 07:18:52.907120 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.907099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"]
Sep 08 07:18:52.984269 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.984194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqd9\" (UniqueName: \"kubernetes.io/projected/b1e84e93-1464-4401-b3d4-a3d2da78bfca-kube-api-access-vkqd9\") pod \"payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc\" (UID: \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:52.984413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.984276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1e84e93-1464-4401-b3d4-a3d2da78bfca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc\" (UID: \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:52.984413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.984310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870d23ee-eb59-41ed-86dd-a24eabc72eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4\" (UID: \"870d23ee-eb59-41ed-86dd-a24eabc72eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:52.984413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.984327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsllg\" (UniqueName: \"kubernetes.io/projected/870d23ee-eb59-41ed-86dd-a24eabc72eed-kube-api-access-nsllg\") pod \"payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4\" (UID: \"870d23ee-eb59-41ed-86dd-a24eabc72eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:52.985077 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.985057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1e84e93-1464-4401-b3d4-a3d2da78bfca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc\" (UID: \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:52.992928 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:52.992912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqd9\" (UniqueName: \"kubernetes.io/projected/b1e84e93-1464-4401-b3d4-a3d2da78bfca-kube-api-access-vkqd9\") pod \"payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc\" (UID: \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:53.085317 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.085282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870d23ee-eb59-41ed-86dd-a24eabc72eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4\" (UID: \"870d23ee-eb59-41ed-86dd-a24eabc72eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:53.085317 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.085319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsllg\" (UniqueName: \"kubernetes.io/projected/870d23ee-eb59-41ed-86dd-a24eabc72eed-kube-api-access-nsllg\") pod \"payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4\" (UID: \"870d23ee-eb59-41ed-86dd-a24eabc72eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:53.086014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.085992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870d23ee-eb59-41ed-86dd-a24eabc72eed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4\" (UID: \"870d23ee-eb59-41ed-86dd-a24eabc72eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:53.092666 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.092642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsllg\" (UniqueName: \"kubernetes.io/projected/870d23ee-eb59-41ed-86dd-a24eabc72eed-kube-api-access-nsllg\") pod \"payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4\" (UID: \"870d23ee-eb59-41ed-86dd-a24eabc72eed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:53.125220 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.125189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:53.205125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.205097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:53.397037 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.397003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"]
Sep 08 07:18:53.399380 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:18:53.399351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e84e93_1464_4401_b3d4_a3d2da78bfca.slice/crio-3598074156d7cb80574d71d63400dd8bdfabad822cffd4b6b6774aa946a4727f WatchSource:0}: Error finding container 3598074156d7cb80574d71d63400dd8bdfabad822cffd4b6b6774aa946a4727f: Status 404 returned error can't find the container with id 3598074156d7cb80574d71d63400dd8bdfabad822cffd4b6b6774aa946a4727f
Sep 08 07:18:53.476520 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.476497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"]
Sep 08 07:18:53.479411 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:18:53.479379    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870d23ee_eb59_41ed_86dd_a24eabc72eed.slice/crio-b08e200ff32033e1c09a6bfb5ef7242ebeb97ab9ef2ed568d7aacdef03acc8f7 WatchSource:0}: Error finding container b08e200ff32033e1c09a6bfb5ef7242ebeb97ab9ef2ed568d7aacdef03acc8f7: Status 404 returned error can't find the container with id b08e200ff32033e1c09a6bfb5ef7242ebeb97ab9ef2ed568d7aacdef03acc8f7
Sep 08 07:18:53.622090 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.622058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc" event={"ID":"b1e84e93-1464-4401-b3d4-a3d2da78bfca","Type":"ContainerStarted","Data":"3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9"}
Sep 08 07:18:53.622278 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.622121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:18:53.622278 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.622143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc" event={"ID":"b1e84e93-1464-4401-b3d4-a3d2da78bfca","Type":"ContainerStarted","Data":"3598074156d7cb80574d71d63400dd8bdfabad822cffd4b6b6774aa946a4727f"}
Sep 08 07:18:53.623849 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.623822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4" event={"ID":"870d23ee-eb59-41ed-86dd-a24eabc72eed","Type":"ContainerStarted","Data":"9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35"}
Sep 08 07:18:53.623962 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.623862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4" event={"ID":"870d23ee-eb59-41ed-86dd-a24eabc72eed","Type":"ContainerStarted","Data":"b08e200ff32033e1c09a6bfb5ef7242ebeb97ab9ef2ed568d7aacdef03acc8f7"}
Sep 08 07:18:53.623962 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.623898    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:18:53.639961 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.639910    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc" podStartSLOduration=1.639891039 podStartE2EDuration="1.639891039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:18:53.637934759 +0000 UTC m=+849.321394914" watchObservedRunningTime="2026-09-08 07:18:53.639891039 +0000 UTC m=+849.323351193"
Sep 08 07:18:53.653277 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.653233    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4" podStartSLOduration=1.653223718 podStartE2EDuration="1.653223718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:18:53.651791043 +0000 UTC m=+849.335251204" watchObservedRunningTime="2026-09-08 07:18:53.653223718 +0000 UTC m=+849.336683870"
Sep 08 07:18:53.910587 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:53.910553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:55.658512 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.658475    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"]
Sep 08 07:18:55.661880 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.661861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"
Sep 08 07:18:55.663773 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.663749    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:18:55.663773 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.663749    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:18:55.670840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.670805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"]
Sep 08 07:18:55.813799 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.813763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cbd945-aec6-439a-9293-5e59405b1aa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt\" (UID: \"56cbd945-aec6-439a-9293-5e59405b1aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"
Sep 08 07:18:55.860645 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.860605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:55.914672 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.914583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cbd945-aec6-439a-9293-5e59405b1aa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt\" (UID: \"56cbd945-aec6-439a-9293-5e59405b1aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"
Sep 08 07:18:55.915248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.915220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cbd945-aec6-439a-9293-5e59405b1aa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt\" (UID: \"56cbd945-aec6-439a-9293-5e59405b1aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"
Sep 08 07:18:55.973504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:55.973453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"
Sep 08 07:18:56.247185 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:56.247105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"]
Sep 08 07:18:56.250406 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:18:56.250376    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cbd945_aec6_439a_9293_5e59405b1aa6.slice/crio-61085fdb5ca7101c3bd8d46f734e3c8d74733c4f172fd0164c7e2f9495578d95 WatchSource:0}: Error finding container 61085fdb5ca7101c3bd8d46f734e3c8d74733c4f172fd0164c7e2f9495578d95: Status 404 returned error can't find the container with id 61085fdb5ca7101c3bd8d46f734e3c8d74733c4f172fd0164c7e2f9495578d95
Sep 08 07:18:56.600725 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:56.600698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:56.641063 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:56.641027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt" event={"ID":"56cbd945-aec6-439a-9293-5e59405b1aa6","Type":"ContainerStarted","Data":"61085fdb5ca7101c3bd8d46f734e3c8d74733c4f172fd0164c7e2f9495578d95"}
Sep 08 07:18:57.385575 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:57.385543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:58.308719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:58.308683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:58.409000 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:58.408976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:58.511333 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:58.511288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:18:59.656654 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:59.656618    2580 generic.go:356] "Generic (PLEG): container finished" podID="56cbd945-aec6-439a-9293-5e59405b1aa6" containerID="5fd4983c812aeef4aabece272956d4d323598f4d3399042bdb15d86f98bcf872" exitCode=0
Sep 08 07:18:59.657042 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:18:59.656690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt" event={"ID":"56cbd945-aec6-439a-9293-5e59405b1aa6","Type":"ContainerDied","Data":"5fd4983c812aeef4aabece272956d4d323598f4d3399042bdb15d86f98bcf872"}
Sep 08 07:19:00.792183 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:00.792154    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"
Sep 08 07:19:00.962624 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:00.962543    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cbd945-aec6-439a-9293-5e59405b1aa6-maas-api-service-ca\") pod \"56cbd945-aec6-439a-9293-5e59405b1aa6\" (UID: \"56cbd945-aec6-439a-9293-5e59405b1aa6\") "
Sep 08 07:19:00.962841 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:00.962820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cbd945-aec6-439a-9293-5e59405b1aa6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56cbd945-aec6-439a-9293-5e59405b1aa6" (UID: "56cbd945-aec6-439a-9293-5e59405b1aa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:19:01.063732 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:01.063692    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cbd945-aec6-439a-9293-5e59405b1aa6-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:19:01.667501 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:01.667451    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"
Sep 08 07:19:01.667709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:01.667476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt" event={"ID":"56cbd945-aec6-439a-9293-5e59405b1aa6","Type":"ContainerDied","Data":"61085fdb5ca7101c3bd8d46f734e3c8d74733c4f172fd0164c7e2f9495578d95"}
Sep 08 07:19:01.667709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:01.667590    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61085fdb5ca7101c3bd8d46f734e3c8d74733c4f172fd0164c7e2f9495578d95"
Sep 08 07:19:04.630647 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:04.630611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:19:04.631098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:04.630982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:19:11.674210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:11.674178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:12.265387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.265350    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:12.294409 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.294376    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"]
Sep 08 07:19:12.294664 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.294638    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl" podUID="998180da-833e-4981-9684-e9ea30c2ad44" containerName="payload-processing" containerID="cri-o://707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e" gracePeriod=30
Sep 08 07:19:12.368106 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.368074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:12.456081 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.456051    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"]
Sep 08 07:19:12.458572 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.458550    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4a1fb56c-55ctt"]
Sep 08 07:19:12.458751 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.458730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:12.521728 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.521640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:12.544718 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.544697    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:19:12.570237 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.570204    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998180da-833e-4981-9684-e9ea30c2ad44-plugins-config-volume\") pod \"998180da-833e-4981-9684-e9ea30c2ad44\" (UID: \"998180da-833e-4981-9684-e9ea30c2ad44\") "
Sep 08 07:19:12.570633 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.570601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998180da-833e-4981-9684-e9ea30c2ad44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "998180da-833e-4981-9684-e9ea30c2ad44" (UID: "998180da-833e-4981-9684-e9ea30c2ad44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:19:12.570751 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.570660    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbq4m\" (UniqueName: \"kubernetes.io/projected/998180da-833e-4981-9684-e9ea30c2ad44-kube-api-access-jbq4m\") pod \"998180da-833e-4981-9684-e9ea30c2ad44\" (UID: \"998180da-833e-4981-9684-e9ea30c2ad44\") "
Sep 08 07:19:12.571014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.570994    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998180da-833e-4981-9684-e9ea30c2ad44-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:19:12.573160 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.573133    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998180da-833e-4981-9684-e9ea30c2ad44-kube-api-access-jbq4m" (OuterVolumeSpecName: "kube-api-access-jbq4m") pod "998180da-833e-4981-9684-e9ea30c2ad44" (UID: "998180da-833e-4981-9684-e9ea30c2ad44"). InnerVolumeSpecName "kube-api-access-jbq4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:19:12.597502 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.597477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:12.672566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.672528    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbq4m\" (UniqueName: \"kubernetes.io/projected/998180da-833e-4981-9684-e9ea30c2ad44-kube-api-access-jbq4m\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:19:12.724890 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.724859    2580 generic.go:356] "Generic (PLEG): container finished" podID="998180da-833e-4981-9684-e9ea30c2ad44" containerID="707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e" exitCode=0
Sep 08 07:19:12.725288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.724904    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"
Sep 08 07:19:12.725288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.724922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl" event={"ID":"998180da-833e-4981-9684-e9ea30c2ad44","Type":"ContainerDied","Data":"707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e"}
Sep 08 07:19:12.725288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.724962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl" event={"ID":"998180da-833e-4981-9684-e9ea30c2ad44","Type":"ContainerDied","Data":"07e8228a9b395fcaade53fb2e95c03731eb995b1788ccc66d7b74389bd5b0933"}
Sep 08 07:19:12.725288 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.724983    2580 scope.go:117] "RemoveContainer" containerID="707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e"
Sep 08 07:19:12.733863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.733714    2580 scope.go:117] "RemoveContainer" containerID="707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e"
Sep 08 07:19:12.734015 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:19:12.733996    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e\": container with ID starting with 707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e not found: ID does not exist" containerID="707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e"
Sep 08 07:19:12.734064 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.734022    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e"} err="failed to get container status \"707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e\": rpc error: code = NotFound desc = could not find container \"707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e\": container with ID starting with 707cf735238366187dfb55b328c99a8a4b2654860dd385f29813fe5fb736b78e not found: ID does not exist"
Sep 08 07:19:12.747670 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.747648    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"]
Sep 08 07:19:12.753065 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.753042    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4a1fb56c-6c66846b7f-wf6tl"]
Sep 08 07:19:12.940585 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.940553    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56cbd945-aec6-439a-9293-5e59405b1aa6" path="/var/lib/kubelet/pods/56cbd945-aec6-439a-9293-5e59405b1aa6/volumes"
Sep 08 07:19:12.940927 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:12.940914    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998180da-833e-4981-9684-e9ea30c2ad44" path="/var/lib/kubelet/pods/998180da-833e-4981-9684-e9ea30c2ad44/volumes"
Sep 08 07:19:13.060848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:13.060818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:13.161149 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:13.161116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:13.266309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:13.266229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:13.365414 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:13.365375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:13.579791 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:13.579762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:14.204770 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:14.204731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:15.839994 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:15.839955    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:16.337583 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:16.337548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:16.436761 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:16.436732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:16.500023 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:16.499986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:16.845619 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:16.845586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:17.093196 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.093146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:17.190329 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.190218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:17.279571 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.279535    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"]
Sep 08 07:19:17.279959 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.279942    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56cbd945-aec6-439a-9293-5e59405b1aa6" containerName="revoke-keys"
Sep 08 07:19:17.280048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.279962    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56cbd945-aec6-439a-9293-5e59405b1aa6" containerName="revoke-keys"
Sep 08 07:19:17.280048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.279991    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998180da-833e-4981-9684-e9ea30c2ad44" containerName="payload-processing"
Sep 08 07:19:17.280048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.280000    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="998180da-833e-4981-9684-e9ea30c2ad44" containerName="payload-processing"
Sep 08 07:19:17.280213 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.280084    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="998180da-833e-4981-9684-e9ea30c2ad44" containerName="payload-processing"
Sep 08 07:19:17.280213 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.280099    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56cbd945-aec6-439a-9293-5e59405b1aa6" containerName="revoke-keys"
Sep 08 07:19:17.284512 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.284490    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:17.286539 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.286516    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c3da4827\""
Sep 08 07:19:17.286661 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.286518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c3da4827-dockercfg-sn26p\""
Sep 08 07:19:17.293509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.293452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"]
Sep 08 07:19:17.313876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.313839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a7204-e346-4076-aef3-79354998f93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz\" (UID: \"766a7204-e346-4076-aef3-79354998f93b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:17.314055 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.313893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjjl\" (UniqueName: \"kubernetes.io/projected/766a7204-e346-4076-aef3-79354998f93b-kube-api-access-lqjjl\") pod \"payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz\" (UID: \"766a7204-e346-4076-aef3-79354998f93b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:17.395667 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.395629    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"]
Sep 08 07:19:17.398913 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.398896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:17.409799 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.409770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"]
Sep 08 07:19:17.414711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.414677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a7204-e346-4076-aef3-79354998f93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz\" (UID: \"766a7204-e346-4076-aef3-79354998f93b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:17.414863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.414764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjjl\" (UniqueName: \"kubernetes.io/projected/766a7204-e346-4076-aef3-79354998f93b-kube-api-access-lqjjl\") pod \"payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz\" (UID: \"766a7204-e346-4076-aef3-79354998f93b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:17.415400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.415375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a7204-e346-4076-aef3-79354998f93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz\" (UID: \"766a7204-e346-4076-aef3-79354998f93b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:17.423049 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.423017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjjl\" (UniqueName: \"kubernetes.io/projected/766a7204-e346-4076-aef3-79354998f93b-kube-api-access-lqjjl\") pod \"payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz\" (UID: \"766a7204-e346-4076-aef3-79354998f93b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:17.515708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.515614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89606e12-9b84-4612-a174-2a0f11526620-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc\" (UID: \"89606e12-9b84-4612-a174-2a0f11526620\") " pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:17.515708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.515657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s47kb\" (UniqueName: \"kubernetes.io/projected/89606e12-9b84-4612-a174-2a0f11526620-kube-api-access-s47kb\") pod \"payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc\" (UID: \"89606e12-9b84-4612-a174-2a0f11526620\") " pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:17.595182 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.595144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:17.616605 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.616563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89606e12-9b84-4612-a174-2a0f11526620-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc\" (UID: \"89606e12-9b84-4612-a174-2a0f11526620\") " pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:17.616605 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.616608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s47kb\" (UniqueName: \"kubernetes.io/projected/89606e12-9b84-4612-a174-2a0f11526620-kube-api-access-s47kb\") pod \"payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc\" (UID: \"89606e12-9b84-4612-a174-2a0f11526620\") " pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:17.617258 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.617232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89606e12-9b84-4612-a174-2a0f11526620-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc\" (UID: \"89606e12-9b84-4612-a174-2a0f11526620\") " pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:17.623817 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.623792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s47kb\" (UniqueName: \"kubernetes.io/projected/89606e12-9b84-4612-a174-2a0f11526620-kube-api-access-s47kb\") pod \"payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc\" (UID: \"89606e12-9b84-4612-a174-2a0f11526620\") " pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:17.710367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.710327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:17.865593 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.865563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"]
Sep 08 07:19:17.868048 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:19:17.868021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766a7204_e346_4076_aef3_79354998f93b.slice/crio-77dcf9f161063389275cb325b9b194b61379a789b879f23ee464b98fd2299125 WatchSource:0}: Error finding container 77dcf9f161063389275cb325b9b194b61379a789b879f23ee464b98fd2299125: Status 404 returned error can't find the container with id 77dcf9f161063389275cb325b9b194b61379a789b879f23ee464b98fd2299125
Sep 08 07:19:17.982742 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:17.982598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"]
Sep 08 07:19:17.984976 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:19:17.984949    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89606e12_9b84_4612_a174_2a0f11526620.slice/crio-e152b0475f268b50548be5550fd4d9eed4232887ea74f8fa08f130c42cc779b6 WatchSource:0}: Error finding container e152b0475f268b50548be5550fd4d9eed4232887ea74f8fa08f130c42cc779b6: Status 404 returned error can't find the container with id e152b0475f268b50548be5550fd4d9eed4232887ea74f8fa08f130c42cc779b6
Sep 08 07:19:18.757001 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:18.756968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc" event={"ID":"89606e12-9b84-4612-a174-2a0f11526620","Type":"ContainerStarted","Data":"5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec"}
Sep 08 07:19:18.757201 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:18.757034    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:18.757201 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:18.757061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc" event={"ID":"89606e12-9b84-4612-a174-2a0f11526620","Type":"ContainerStarted","Data":"e152b0475f268b50548be5550fd4d9eed4232887ea74f8fa08f130c42cc779b6"}
Sep 08 07:19:18.758547 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:18.758523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz" event={"ID":"766a7204-e346-4076-aef3-79354998f93b","Type":"ContainerStarted","Data":"cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be"}
Sep 08 07:19:18.758659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:18.758561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz" event={"ID":"766a7204-e346-4076-aef3-79354998f93b","Type":"ContainerStarted","Data":"77dcf9f161063389275cb325b9b194b61379a789b879f23ee464b98fd2299125"}
Sep 08 07:19:18.758659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:18.758646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:18.775550 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:18.775507    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc" podStartSLOduration=1.77549455 podStartE2EDuration="1.77549455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:19:18.773410768 +0000 UTC m=+874.456870922" watchObservedRunningTime="2026-09-08 07:19:18.77549455 +0000 UTC m=+874.458954704"
Sep 08 07:19:18.788404 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:18.788365    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz" podStartSLOduration=1.788352028 podStartE2EDuration="1.788352028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:19:18.787885838 +0000 UTC m=+874.471345994" watchObservedRunningTime="2026-09-08 07:19:18.788352028 +0000 UTC m=+874.471812181"
Sep 08 07:19:21.938297 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:21.938267    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"]
Sep 08 07:19:21.941945 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:21.941924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:21.943899 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:21.943880    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-f2324022-kserve-self-signed-certs\""
Sep 08 07:19:21.949714 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:21.949693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"]
Sep 08 07:19:21.960867 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:21.960838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:22.058115 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.058069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tmp-dir\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.058296 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.058120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rgx\" (UniqueName: \"kubernetes.io/projected/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-kube-api-access-q2rgx\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.058296 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.058234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-dshm\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.058393 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.058310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tls-certs\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.058572 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.058548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-model-cache\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.058666 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.058640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-home\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.159427 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tls-certs\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.159599 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-model-cache\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.159599 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-home\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.159734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tmp-dir\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.159734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rgx\" (UniqueName: \"kubernetes.io/projected/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-kube-api-access-q2rgx\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.159734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-dshm\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.159944 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-model-cache\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.160010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-home\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.160010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.159995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tmp-dir\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.161737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.161717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-dshm\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.161904 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.161886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tls-certs\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.167008 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.166986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rgx\" (UniqueName: \"kubernetes.io/projected/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-kube-api-access-q2rgx\") pod \"e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") " pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.251595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.251510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:22.521261 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.521233    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"]
Sep 08 07:19:22.523350 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:19:22.523319    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22d0e4d4_5e0f_4e9a_83d2_1ad9a8d1131c.slice/crio-96e369444050802453e25b9eae798ff5aa144f2690db52b6000dca4cd8aef8de WatchSource:0}: Error finding container 96e369444050802453e25b9eae798ff5aa144f2690db52b6000dca4cd8aef8de: Status 404 returned error can't find the container with id 96e369444050802453e25b9eae798ff5aa144f2690db52b6000dca4cd8aef8de
Sep 08 07:19:22.780200 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.780090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd" event={"ID":"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c","Type":"ContainerStarted","Data":"4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636"}
Sep 08 07:19:22.780200 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.780154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd" event={"ID":"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c","Type":"ContainerStarted","Data":"96e369444050802453e25b9eae798ff5aa144f2690db52b6000dca4cd8aef8de"}
Sep 08 07:19:22.796410 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:22.796354    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd" podStartSLOduration=1.7963236550000001 podStartE2EDuration="1.796323655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:19:22.794654624 +0000 UTC m=+878.478114776" watchObservedRunningTime="2026-09-08 07:19:22.796323655 +0000 UTC m=+878.479783807"
Sep 08 07:19:29.765383 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:29.765347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:29.765764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:29.765430    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:32.252542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:32.252500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:32.253023 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:32.252556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:32.265066 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:32.265036    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:32.844310 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:32.844279    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:19:49.564008 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:49.563969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:51.601668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:51.601635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:57.274467 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:57.274424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:58.179331 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.179293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:58.266388 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.266352    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"]
Sep 08 07:19:58.267073 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.266655    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz" podUID="766a7204-e346-4076-aef3-79354998f93b" containerName="payload-pre-processing" containerID="cri-o://cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be" gracePeriod=30
Sep 08 07:19:58.268788 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.268763    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"]
Sep 08 07:19:58.268985 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.268961    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc" podUID="89606e12-9b84-4612-a174-2a0f11526620" containerName="payload-processing" containerID="cri-o://5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec" gracePeriod=30
Sep 08 07:19:58.295084 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.295059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:58.396892 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.396853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:58.497482 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.497438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:58.542861 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.542836    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:58.545883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.545856    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:58.569413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.569377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:58.597498 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.597449    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqjjl\" (UniqueName: \"kubernetes.io/projected/766a7204-e346-4076-aef3-79354998f93b-kube-api-access-lqjjl\") pod \"766a7204-e346-4076-aef3-79354998f93b\" (UID: \"766a7204-e346-4076-aef3-79354998f93b\") "
Sep 08 07:19:58.597667 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.597546    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a7204-e346-4076-aef3-79354998f93b-plugins-config-volume\") pod \"766a7204-e346-4076-aef3-79354998f93b\" (UID: \"766a7204-e346-4076-aef3-79354998f93b\") "
Sep 08 07:19:58.597667 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.597566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s47kb\" (UniqueName: \"kubernetes.io/projected/89606e12-9b84-4612-a174-2a0f11526620-kube-api-access-s47kb\") pod \"89606e12-9b84-4612-a174-2a0f11526620\" (UID: \"89606e12-9b84-4612-a174-2a0f11526620\") "
Sep 08 07:19:58.597667 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.597585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89606e12-9b84-4612-a174-2a0f11526620-plugins-config-volume\") pod \"89606e12-9b84-4612-a174-2a0f11526620\" (UID: \"89606e12-9b84-4612-a174-2a0f11526620\") "
Sep 08 07:19:58.598018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.597982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766a7204-e346-4076-aef3-79354998f93b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "766a7204-e346-4076-aef3-79354998f93b" (UID: "766a7204-e346-4076-aef3-79354998f93b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:19:58.598018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.597982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89606e12-9b84-4612-a174-2a0f11526620-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89606e12-9b84-4612-a174-2a0f11526620" (UID: "89606e12-9b84-4612-a174-2a0f11526620"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:19:58.599551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.599525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766a7204-e346-4076-aef3-79354998f93b-kube-api-access-lqjjl" (OuterVolumeSpecName: "kube-api-access-lqjjl") pod "766a7204-e346-4076-aef3-79354998f93b" (UID: "766a7204-e346-4076-aef3-79354998f93b"). InnerVolumeSpecName "kube-api-access-lqjjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:19:58.599771 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.599742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89606e12-9b84-4612-a174-2a0f11526620-kube-api-access-s47kb" (OuterVolumeSpecName: "kube-api-access-s47kb") pod "89606e12-9b84-4612-a174-2a0f11526620" (UID: "89606e12-9b84-4612-a174-2a0f11526620"). InnerVolumeSpecName "kube-api-access-s47kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:19:58.686674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.686644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:58.698915 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.698848    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766a7204-e346-4076-aef3-79354998f93b-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:19:58.698915 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.698874    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s47kb\" (UniqueName: \"kubernetes.io/projected/89606e12-9b84-4612-a174-2a0f11526620-kube-api-access-s47kb\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:19:58.698915 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.698884    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89606e12-9b84-4612-a174-2a0f11526620-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:19:58.698915 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.698893    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjjl\" (UniqueName: \"kubernetes.io/projected/766a7204-e346-4076-aef3-79354998f93b-kube-api-access-lqjjl\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:19:58.962861 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.962833    2580 generic.go:356] "Generic (PLEG): container finished" podID="89606e12-9b84-4612-a174-2a0f11526620" containerID="5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec" exitCode=0
Sep 08 07:19:58.962981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.962870    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"
Sep 08 07:19:58.962981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.962899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc" event={"ID":"89606e12-9b84-4612-a174-2a0f11526620","Type":"ContainerDied","Data":"5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec"}
Sep 08 07:19:58.962981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.962942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc" event={"ID":"89606e12-9b84-4612-a174-2a0f11526620","Type":"ContainerDied","Data":"e152b0475f268b50548be5550fd4d9eed4232887ea74f8fa08f130c42cc779b6"}
Sep 08 07:19:58.962981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.962965    2580 scope.go:117] "RemoveContainer" containerID="5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec"
Sep 08 07:19:58.964192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.964170    2580 generic.go:356] "Generic (PLEG): container finished" podID="766a7204-e346-4076-aef3-79354998f93b" containerID="cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be" exitCode=0
Sep 08 07:19:58.964286 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.964211    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"
Sep 08 07:19:58.964286 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.964252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz" event={"ID":"766a7204-e346-4076-aef3-79354998f93b","Type":"ContainerDied","Data":"cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be"}
Sep 08 07:19:58.964367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.964294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz" event={"ID":"766a7204-e346-4076-aef3-79354998f93b","Type":"ContainerDied","Data":"77dcf9f161063389275cb325b9b194b61379a789b879f23ee464b98fd2299125"}
Sep 08 07:19:58.972281 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.972265    2580 scope.go:117] "RemoveContainer" containerID="5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec"
Sep 08 07:19:58.972595 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:19:58.972576    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec\": container with ID starting with 5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec not found: ID does not exist" containerID="5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec"
Sep 08 07:19:58.972646 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.972603    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec"} err="failed to get container status \"5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec\": rpc error: code = NotFound desc = could not find container \"5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec\": container with ID starting with 5a8d429d96945392d35ac1fbf04991cd004eedf6be1c15c13183ac28c38fadec not found: ID does not exist"
Sep 08 07:19:58.972646 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.972618    2580 scope.go:117] "RemoveContainer" containerID="cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be"
Sep 08 07:19:58.980492 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.980452    2580 scope.go:117] "RemoveContainer" containerID="cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be"
Sep 08 07:19:58.980786 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:19:58.980766    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be\": container with ID starting with cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be not found: ID does not exist" containerID="cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be"
Sep 08 07:19:58.980826 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.980792    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be"} err="failed to get container status \"cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be\": rpc error: code = NotFound desc = could not find container \"cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be\": container with ID starting with cd7e5fd96b925de3d6f39fb2018e47cf2e981deddf3a6e32c92ad19cdba519be not found: ID does not exist"
Sep 08 07:19:58.981522 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.981505    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"]
Sep 08 07:19:58.986447 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.986426    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c3da4827-77f465c998-4jdwc"]
Sep 08 07:19:58.997109 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:58.997081    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"]
Sep 08 07:19:59.003492 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:59.003472    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c3da4827-78c95df78b-5fwzz"]
Sep 08 07:19:59.106739 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:59.106704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:59.436769 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:59.436736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:19:59.586750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:19:59.586715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:00.940937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:00.940900    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="766a7204-e346-4076-aef3-79354998f93b" path="/var/lib/kubelet/pods/766a7204-e346-4076-aef3-79354998f93b/volumes"
Sep 08 07:20:00.943171 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:00.941257    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89606e12-9b84-4612-a174-2a0f11526620" path="/var/lib/kubelet/pods/89606e12-9b84-4612-a174-2a0f11526620/volumes"
Sep 08 07:20:01.538053 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:01.538015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:01.637277 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:01.637245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:02.766660 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:02.766624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:02.860954 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:02.860920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:03.509819 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.509779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:03.617145 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.617112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:03.806837 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.806759    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"]
Sep 08 07:20:03.807187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.807117    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766a7204-e346-4076-aef3-79354998f93b" containerName="payload-pre-processing"
Sep 08 07:20:03.807187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.807127    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="766a7204-e346-4076-aef3-79354998f93b" containerName="payload-pre-processing"
Sep 08 07:20:03.807187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.807136    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89606e12-9b84-4612-a174-2a0f11526620" containerName="payload-processing"
Sep 08 07:20:03.807187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.807141    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89606e12-9b84-4612-a174-2a0f11526620" containerName="payload-processing"
Sep 08 07:20:03.807325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.807193    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89606e12-9b84-4612-a174-2a0f11526620" containerName="payload-processing"
Sep 08 07:20:03.807325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.807204    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="766a7204-e346-4076-aef3-79354998f93b" containerName="payload-pre-processing"
Sep 08 07:20:03.810250 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.810231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:03.812316 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.812293    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-576ac2af\""
Sep 08 07:20:03.812607 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.812587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-576ac2af-dockercfg-46zcc\""
Sep 08 07:20:03.821491 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.821446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"]
Sep 08 07:20:03.954284 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.954244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e274f6-87c8-48a2-a1f9-3447b288522f-plugins-config-volume\") pod \"payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh\" (UID: \"b8e274f6-87c8-48a2-a1f9-3447b288522f\") " pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:03.954485 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:03.954376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7qw\" (UniqueName: \"kubernetes.io/projected/b8e274f6-87c8-48a2-a1f9-3447b288522f-kube-api-access-fc7qw\") pod \"payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh\" (UID: \"b8e274f6-87c8-48a2-a1f9-3447b288522f\") " pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:04.055741 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.055699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e274f6-87c8-48a2-a1f9-3447b288522f-plugins-config-volume\") pod \"payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh\" (UID: \"b8e274f6-87c8-48a2-a1f9-3447b288522f\") " pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:04.055940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.055797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7qw\" (UniqueName: \"kubernetes.io/projected/b8e274f6-87c8-48a2-a1f9-3447b288522f-kube-api-access-fc7qw\") pod \"payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh\" (UID: \"b8e274f6-87c8-48a2-a1f9-3447b288522f\") " pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:04.056493 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.056450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e274f6-87c8-48a2-a1f9-3447b288522f-plugins-config-volume\") pod \"payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh\" (UID: \"b8e274f6-87c8-48a2-a1f9-3447b288522f\") " pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:04.064779 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.064717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7qw\" (UniqueName: \"kubernetes.io/projected/b8e274f6-87c8-48a2-a1f9-3447b288522f-kube-api-access-fc7qw\") pod \"payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh\" (UID: \"b8e274f6-87c8-48a2-a1f9-3447b288522f\") " pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:04.122425 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.122376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:04.410837 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.410703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"]
Sep 08 07:20:04.416918 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:20:04.416880    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e274f6_87c8_48a2_a1f9_3447b288522f.slice/crio-7f9c8ee74553d1e82240f88c4c4f905f0ce3bc23e152b13e9d936c647b91bd14 WatchSource:0}: Error finding container 7f9c8ee74553d1e82240f88c4c4f905f0ce3bc23e152b13e9d936c647b91bd14: Status 404 returned error can't find the container with id 7f9c8ee74553d1e82240f88c4c4f905f0ce3bc23e152b13e9d936c647b91bd14
Sep 08 07:20:04.996863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.996823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh" event={"ID":"b8e274f6-87c8-48a2-a1f9-3447b288522f","Type":"ContainerStarted","Data":"1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c"}
Sep 08 07:20:04.997339 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.996882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh" event={"ID":"b8e274f6-87c8-48a2-a1f9-3447b288522f","Type":"ContainerStarted","Data":"7f9c8ee74553d1e82240f88c4c4f905f0ce3bc23e152b13e9d936c647b91bd14"}
Sep 08 07:20:04.997339 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:04.996927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:05.028755 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:05.028699    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh" podStartSLOduration=2.02867982 podStartE2EDuration="2.02867982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:20:05.026265897 +0000 UTC m=+920.709726050" watchObservedRunningTime="2026-09-08 07:20:05.02867982 +0000 UTC m=+920.712139973"
Sep 08 07:20:16.005159 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:16.005119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:26.116405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:26.116373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:26.865603 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:26.865569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:27.142509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.142419    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"]
Sep 08 07:20:27.146069 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.146049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"
Sep 08 07:20:27.147938 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.147916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:20:27.148105 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.148088    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:20:27.151762 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.151734    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"]
Sep 08 07:20:27.176043 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.176007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f72268-18aa-4378-9a36-004aaf2bea13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-576ac2af-xgtns\" (UID: \"21f72268-18aa-4378-9a36-004aaf2bea13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"
Sep 08 07:20:27.277255 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.277216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f72268-18aa-4378-9a36-004aaf2bea13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-576ac2af-xgtns\" (UID: \"21f72268-18aa-4378-9a36-004aaf2bea13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"
Sep 08 07:20:27.277850 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.277829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f72268-18aa-4378-9a36-004aaf2bea13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-576ac2af-xgtns\" (UID: \"21f72268-18aa-4378-9a36-004aaf2bea13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"
Sep 08 07:20:27.457400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.457312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"
Sep 08 07:20:27.728778 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:27.728750    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"]
Sep 08 07:20:27.731002 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:20:27.730973    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f72268_18aa_4378_9a36_004aaf2bea13.slice/crio-295220f90398ac4edb0d479e58c1e0f64a5463d6ce7f50b20d3161f53af94c64 WatchSource:0}: Error finding container 295220f90398ac4edb0d479e58c1e0f64a5463d6ce7f50b20d3161f53af94c64: Status 404 returned error can't find the container with id 295220f90398ac4edb0d479e58c1e0f64a5463d6ce7f50b20d3161f53af94c64
Sep 08 07:20:28.109399 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:28.109364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns" event={"ID":"21f72268-18aa-4378-9a36-004aaf2bea13","Type":"ContainerStarted","Data":"5a82413241b13259ce45e2272db8c63e52e85d9b188eb338604e90a66d2b90bb"}
Sep 08 07:20:28.109587 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:28.109418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns" event={"ID":"21f72268-18aa-4378-9a36-004aaf2bea13","Type":"ContainerStarted","Data":"295220f90398ac4edb0d479e58c1e0f64a5463d6ce7f50b20d3161f53af94c64"}
Sep 08 07:20:28.125477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:28.125419    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns" podStartSLOduration=1.125401383 podStartE2EDuration="1.125401383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:20:28.123532677 +0000 UTC m=+943.806992832" watchObservedRunningTime="2026-09-08 07:20:28.125401383 +0000 UTC m=+943.808861575"
Sep 08 07:20:29.115883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:29.115846    2580 generic.go:356] "Generic (PLEG): container finished" podID="21f72268-18aa-4378-9a36-004aaf2bea13" containerID="5a82413241b13259ce45e2272db8c63e52e85d9b188eb338604e90a66d2b90bb" exitCode=0
Sep 08 07:20:29.116268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:29.115931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns" event={"ID":"21f72268-18aa-4378-9a36-004aaf2bea13","Type":"ContainerDied","Data":"5a82413241b13259ce45e2272db8c63e52e85d9b188eb338604e90a66d2b90bb"}
Sep 08 07:20:30.247869 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:30.247844    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"
Sep 08 07:20:30.306725 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:30.306688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f72268-18aa-4378-9a36-004aaf2bea13-maas-api-service-ca\") pod \"21f72268-18aa-4378-9a36-004aaf2bea13\" (UID: \"21f72268-18aa-4378-9a36-004aaf2bea13\") "
Sep 08 07:20:30.307050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:30.307027    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f72268-18aa-4378-9a36-004aaf2bea13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21f72268-18aa-4378-9a36-004aaf2bea13" (UID: "21f72268-18aa-4378-9a36-004aaf2bea13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:20:30.408268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:30.408174    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f72268-18aa-4378-9a36-004aaf2bea13-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:20:31.126368 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:31.126343    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"
Sep 08 07:20:31.126585 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:31.126343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns" event={"ID":"21f72268-18aa-4378-9a36-004aaf2bea13","Type":"ContainerDied","Data":"295220f90398ac4edb0d479e58c1e0f64a5463d6ce7f50b20d3161f53af94c64"}
Sep 08 07:20:31.126585 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:31.126476    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="295220f90398ac4edb0d479e58c1e0f64a5463d6ce7f50b20d3161f53af94c64"
Sep 08 07:20:31.982859 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:31.982825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:33.191648 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.191602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:33.215451 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.215418    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"]
Sep 08 07:20:33.215734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.215703    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh" podUID="b8e274f6-87c8-48a2-a1f9-3447b288522f" containerName="payload-processing" containerID="cri-o://1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c" gracePeriod=30
Sep 08 07:20:33.307941 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.307909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:33.392379 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.392286    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"]
Sep 08 07:20:33.396234 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.395854    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576ac2af-xgtns"]
Sep 08 07:20:33.397071 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.396674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:33.457723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.457694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:33.465698 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.465677    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:33.538959 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.538929    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc7qw\" (UniqueName: \"kubernetes.io/projected/b8e274f6-87c8-48a2-a1f9-3447b288522f-kube-api-access-fc7qw\") pod \"b8e274f6-87c8-48a2-a1f9-3447b288522f\" (UID: \"b8e274f6-87c8-48a2-a1f9-3447b288522f\") "
Sep 08 07:20:33.538959 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.538964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e274f6-87c8-48a2-a1f9-3447b288522f-plugins-config-volume\") pod \"b8e274f6-87c8-48a2-a1f9-3447b288522f\" (UID: \"b8e274f6-87c8-48a2-a1f9-3447b288522f\") "
Sep 08 07:20:33.539351 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.539329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e274f6-87c8-48a2-a1f9-3447b288522f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8e274f6-87c8-48a2-a1f9-3447b288522f" (UID: "b8e274f6-87c8-48a2-a1f9-3447b288522f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:20:33.541012 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.540988    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e274f6-87c8-48a2-a1f9-3447b288522f-kube-api-access-fc7qw" (OuterVolumeSpecName: "kube-api-access-fc7qw") pod "b8e274f6-87c8-48a2-a1f9-3447b288522f" (UID: "b8e274f6-87c8-48a2-a1f9-3447b288522f"). InnerVolumeSpecName "kube-api-access-fc7qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:20:33.640364 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.640316    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc7qw\" (UniqueName: \"kubernetes.io/projected/b8e274f6-87c8-48a2-a1f9-3447b288522f-kube-api-access-fc7qw\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:20:33.640364 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:33.640358    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e274f6-87c8-48a2-a1f9-3447b288522f-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:20:34.142205 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.142175    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8e274f6-87c8-48a2-a1f9-3447b288522f" containerID="1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c" exitCode=0
Sep 08 07:20:34.142400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.142219    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"
Sep 08 07:20:34.142400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.142261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh" event={"ID":"b8e274f6-87c8-48a2-a1f9-3447b288522f","Type":"ContainerDied","Data":"1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c"}
Sep 08 07:20:34.142400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.142309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh" event={"ID":"b8e274f6-87c8-48a2-a1f9-3447b288522f","Type":"ContainerDied","Data":"7f9c8ee74553d1e82240f88c4c4f905f0ce3bc23e152b13e9d936c647b91bd14"}
Sep 08 07:20:34.142400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.142336    2580 scope.go:117] "RemoveContainer" containerID="1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c"
Sep 08 07:20:34.151536 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.151520    2580 scope.go:117] "RemoveContainer" containerID="1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c"
Sep 08 07:20:34.151801 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:20:34.151784    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c\": container with ID starting with 1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c not found: ID does not exist" containerID="1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c"
Sep 08 07:20:34.151850 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.151810    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c"} err="failed to get container status \"1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c\": rpc error: code = NotFound desc = could not find container \"1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c\": container with ID starting with 1724b6ac6a8d8d0783c9f08305b549537663194b7b5c5568b6c8f1680d9e9d7c not found: ID does not exist"
Sep 08 07:20:34.163357 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.163332    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"]
Sep 08 07:20:34.167227 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.167205    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576ac2af-8585ccc9b5-qgrnh"]
Sep 08 07:20:34.462333 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.462244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:34.872920 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.872887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:34.942014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.941985    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f72268-18aa-4378-9a36-004aaf2bea13" path="/var/lib/kubelet/pods/21f72268-18aa-4378-9a36-004aaf2bea13/volumes"
Sep 08 07:20:34.942345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:34.942332    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8e274f6-87c8-48a2-a1f9-3447b288522f" path="/var/lib/kubelet/pods/b8e274f6-87c8-48a2-a1f9-3447b288522f/volumes"
Sep 08 07:20:35.138658 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:35.138584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:35.434663 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:35.434580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:36.949543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.949510    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"]
Sep 08 07:20:36.950065 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.950037    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f72268-18aa-4378-9a36-004aaf2bea13" containerName="revoke-keys"
Sep 08 07:20:36.950065 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.950067    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f72268-18aa-4378-9a36-004aaf2bea13" containerName="revoke-keys"
Sep 08 07:20:36.950212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.950105    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e274f6-87c8-48a2-a1f9-3447b288522f" containerName="payload-processing"
Sep 08 07:20:36.950212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.950115    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e274f6-87c8-48a2-a1f9-3447b288522f" containerName="payload-processing"
Sep 08 07:20:36.950306 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.950210    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8e274f6-87c8-48a2-a1f9-3447b288522f" containerName="payload-processing"
Sep 08 07:20:36.950306 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.950221    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f72268-18aa-4378-9a36-004aaf2bea13" containerName="revoke-keys"
Sep 08 07:20:36.954384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.954355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:36.957161 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.957139    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-620112\"/\"kube-root-ca.crt\""
Sep 08 07:20:36.957810 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.957501    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-620112\"/\"test-model-620112-kserve-self-signed-certs\""
Sep 08 07:20:36.958019 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.957554    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-620112\"/\"default-dockercfg-q4ccs\""
Sep 08 07:20:36.958171 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.957602    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-620112\"/\"openshift-service-ca.crt\""
Sep 08 07:20:36.963782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:36.963715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"]
Sep 08 07:20:37.073227 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.073183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-tmp-dir\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.073227 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.073227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-home\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.073432 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.073246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-model-cache\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.073432 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.073371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c92dec-10c8-405c-8242-d8e80cffe6ce-tls-certs\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.073432 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.073413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-dshm\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.073554 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.073508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vgp\" (UniqueName: \"kubernetes.io/projected/13c92dec-10c8-405c-8242-d8e80cffe6ce-kube-api-access-r9vgp\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.174830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.174789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c92dec-10c8-405c-8242-d8e80cffe6ce-tls-certs\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.174976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.174838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-dshm\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.174976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.174870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vgp\" (UniqueName: \"kubernetes.io/projected/13c92dec-10c8-405c-8242-d8e80cffe6ce-kube-api-access-r9vgp\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.174976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.174902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-tmp-dir\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.174976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.174921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-home\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.174976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.174937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-model-cache\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.175370 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.175334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-tmp-dir\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.175517 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.175371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-model-cache\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.175517 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.175403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-home\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.177123 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.177101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-dshm\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.177394 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.177376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c92dec-10c8-405c-8242-d8e80cffe6ce-tls-certs\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.183316 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.183296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vgp\" (UniqueName: \"kubernetes.io/projected/13c92dec-10c8-405c-8242-d8e80cffe6ce-kube-api-access-r9vgp\") pod \"test-model-620112-kserve-86b89cb6b9-7tlm8\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") " pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.237676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.237594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:37.266734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.266699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:37.536937 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.536912    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"]
Sep 08 07:20:37.539489 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:20:37.539438    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13c92dec_10c8_405c_8242_d8e80cffe6ce.slice/crio-7012057756eb3142f0b0b31918f97f5780ddfebdc61223bd3eaca61127be26bf WatchSource:0}: Error finding container 7012057756eb3142f0b0b31918f97f5780ddfebdc61223bd3eaca61127be26bf: Status 404 returned error can't find the container with id 7012057756eb3142f0b0b31918f97f5780ddfebdc61223bd3eaca61127be26bf
Sep 08 07:20:37.706634 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.706603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:37.806924 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:37.806838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:38.164454 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:38.164417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8" event={"ID":"13c92dec-10c8-405c-8242-d8e80cffe6ce","Type":"ContainerStarted","Data":"f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae"}
Sep 08 07:20:38.164855 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:38.164481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8" event={"ID":"13c92dec-10c8-405c-8242-d8e80cffe6ce","Type":"ContainerStarted","Data":"7012057756eb3142f0b0b31918f97f5780ddfebdc61223bd3eaca61127be26bf"}
Sep 08 07:20:38.180446 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:38.180394    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8" podStartSLOduration=2.180378744 podStartE2EDuration="2.180378744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:20:38.179778941 +0000 UTC m=+953.863239099" watchObservedRunningTime="2026-09-08 07:20:38.180378744 +0000 UTC m=+953.863838896"
Sep 08 07:20:38.917017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:38.916982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:39.058639 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.058600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:39.416669 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.416635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:39.523441 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.523410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:39.545825 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.545794    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"]
Sep 08 07:20:39.549864 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.549841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:39.552018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.551994    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-da54544b-dockercfg-vg9sv\""
Sep 08 07:20:39.552018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.551995    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-da54544b\""
Sep 08 07:20:39.559687 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.559663    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"]
Sep 08 07:20:39.619956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.619919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:39.699154 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.699051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbq5v\" (UniqueName: \"kubernetes.io/projected/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-kube-api-access-wbq5v\") pod \"payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq\" (UID: \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\") " pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:39.699154 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.699109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-plugins-config-volume\") pod \"payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq\" (UID: \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\") " pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:39.800274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.800237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbq5v\" (UniqueName: \"kubernetes.io/projected/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-kube-api-access-wbq5v\") pod \"payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq\" (UID: \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\") " pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:39.800274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.800279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-plugins-config-volume\") pod \"payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq\" (UID: \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\") " pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:39.801016 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.800997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-plugins-config-volume\") pod \"payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq\" (UID: \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\") " pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:39.809160 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.809130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbq5v\" (UniqueName: \"kubernetes.io/projected/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-kube-api-access-wbq5v\") pod \"payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq\" (UID: \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\") " pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:39.861727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:39.861678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:40.138033 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:40.138004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"]
Sep 08 07:20:40.140423 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:20:40.140386    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01bdd106_6c50_40ff_9afb_f7b3e5d09ca6.slice/crio-8999f8b2f6629df0a31e7c68b0257707c87df019463ed53f0a3eab3f8d90de1c WatchSource:0}: Error finding container 8999f8b2f6629df0a31e7c68b0257707c87df019463ed53f0a3eab3f8d90de1c: Status 404 returned error can't find the container with id 8999f8b2f6629df0a31e7c68b0257707c87df019463ed53f0a3eab3f8d90de1c
Sep 08 07:20:40.175691 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:40.175657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq" event={"ID":"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6","Type":"ContainerStarted","Data":"8999f8b2f6629df0a31e7c68b0257707c87df019463ed53f0a3eab3f8d90de1c"}
Sep 08 07:20:41.181719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:41.181682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq" event={"ID":"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6","Type":"ContainerStarted","Data":"847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9"}
Sep 08 07:20:41.182111 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:41.181813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:41.201180 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:41.201128    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq" podStartSLOduration=2.201111224 podStartE2EDuration="2.201111224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:20:41.199004124 +0000 UTC m=+956.882464294" watchObservedRunningTime="2026-09-08 07:20:41.201111224 +0000 UTC m=+956.884571415"
Sep 08 07:20:42.122569 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.122538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:42.566925 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.566841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:42.599329 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.599299    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"]
Sep 08 07:20:42.599589 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.599560    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq" podUID="3befb5eb-9437-41c3-bb59-e5b3884c3df4" containerName="payload-processing" containerID="cri-o://9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925" gracePeriod=30
Sep 08 07:20:42.631733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.631692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:42.711583 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.711548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:42.780850 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.780805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:42.846395 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.846354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:42.848711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.848689    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:20:42.931674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.931641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befb5eb-9437-41c3-bb59-e5b3884c3df4-plugins-config-volume\") pod \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\" (UID: \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\") "
Sep 08 07:20:42.931854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.931712    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7k5n\" (UniqueName: \"kubernetes.io/projected/3befb5eb-9437-41c3-bb59-e5b3884c3df4-kube-api-access-b7k5n\") pod \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\" (UID: \"3befb5eb-9437-41c3-bb59-e5b3884c3df4\") "
Sep 08 07:20:42.932016 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.931991    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3befb5eb-9437-41c3-bb59-e5b3884c3df4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3befb5eb-9437-41c3-bb59-e5b3884c3df4" (UID: "3befb5eb-9437-41c3-bb59-e5b3884c3df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:20:42.933963 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:42.933937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3befb5eb-9437-41c3-bb59-e5b3884c3df4-kube-api-access-b7k5n" (OuterVolumeSpecName: "kube-api-access-b7k5n") pod "3befb5eb-9437-41c3-bb59-e5b3884c3df4" (UID: "3befb5eb-9437-41c3-bb59-e5b3884c3df4"). InnerVolumeSpecName "kube-api-access-b7k5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:20:43.032782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.032750    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befb5eb-9437-41c3-bb59-e5b3884c3df4-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:20:43.032782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.032777    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7k5n\" (UniqueName: \"kubernetes.io/projected/3befb5eb-9437-41c3-bb59-e5b3884c3df4-kube-api-access-b7k5n\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:20:43.112088 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.112002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:43.193084 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.193047    2580 generic.go:356] "Generic (PLEG): container finished" podID="3befb5eb-9437-41c3-bb59-e5b3884c3df4" containerID="9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925" exitCode=0
Sep 08 07:20:43.193252 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.193089    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"
Sep 08 07:20:43.193252 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.193136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq" event={"ID":"3befb5eb-9437-41c3-bb59-e5b3884c3df4","Type":"ContainerDied","Data":"9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925"}
Sep 08 07:20:43.193252 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.193188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq" event={"ID":"3befb5eb-9437-41c3-bb59-e5b3884c3df4","Type":"ContainerDied","Data":"b0eceb56c58586eb65011b75f1592dfe7d7b04dcde6b71f83389a8a3a0754407"}
Sep 08 07:20:43.193252 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.193211    2580 scope.go:117] "RemoveContainer" containerID="9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925"
Sep 08 07:20:43.202250 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.202231    2580 scope.go:117] "RemoveContainer" containerID="9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925"
Sep 08 07:20:43.202552 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:20:43.202521    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925\": container with ID starting with 9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925 not found: ID does not exist" containerID="9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925"
Sep 08 07:20:43.202643 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.202549    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925"} err="failed to get container status \"9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925\": rpc error: code = NotFound desc = could not find container \"9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925\": container with ID starting with 9ce41989c5dd02a49e7cba6d42e3140fb5e6620f2991a769921f6a2bca35e925 not found: ID does not exist"
Sep 08 07:20:43.208035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.208013    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"]
Sep 08 07:20:43.212401 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:43.212378    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-638d2d-7d646f7c95-xtjcq"]
Sep 08 07:20:44.941303 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:44.941269    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3befb5eb-9437-41c3-bb59-e5b3884c3df4" path="/var/lib/kubelet/pods/3befb5eb-9437-41c3-bb59-e5b3884c3df4/volumes"
Sep 08 07:20:47.266889 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:47.266845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:47.267361 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:47.266910    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:47.279966 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:47.279916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:48.195852 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:48.195814    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:20:48.229205 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:48.229172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:20:52.189110 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:52.189080    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:20:59.885164 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:20:59.885123    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:09.604830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.604743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:09.627895 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.627857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"]
Sep 08 07:21:09.628172 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.628146    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq" podUID="01bdd106-6c50-40ff-9afb-f7b3e5d09ca6" containerName="payload-processing" containerID="cri-o://847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9" gracePeriod=30
Sep 08 07:21:09.668420 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.668387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:09.766879 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.766845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:09.868710 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.868678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:09.881388 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.881366    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:21:09.940444 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.940416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:09.972512 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.972487    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-plugins-config-volume\") pod \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\" (UID: \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\") "
Sep 08 07:21:09.972680 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.972576    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbq5v\" (UniqueName: \"kubernetes.io/projected/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-kube-api-access-wbq5v\") pod \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\" (UID: \"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6\") "
Sep 08 07:21:09.972901 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.972872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01bdd106-6c50-40ff-9afb-f7b3e5d09ca6" (UID: "01bdd106-6c50-40ff-9afb-f7b3e5d09ca6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:21:09.974607 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:09.974582    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-kube-api-access-wbq5v" (OuterVolumeSpecName: "kube-api-access-wbq5v") pod "01bdd106-6c50-40ff-9afb-f7b3e5d09ca6" (UID: "01bdd106-6c50-40ff-9afb-f7b3e5d09ca6"). InnerVolumeSpecName "kube-api-access-wbq5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:21:10.073954 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.073910    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbq5v\" (UniqueName: \"kubernetes.io/projected/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-kube-api-access-wbq5v\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:21:10.073954 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.073945    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:21:10.328556 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.328521    2580 generic.go:356] "Generic (PLEG): container finished" podID="01bdd106-6c50-40ff-9afb-f7b3e5d09ca6" containerID="847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9" exitCode=0
Sep 08 07:21:10.328730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.328566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq" event={"ID":"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6","Type":"ContainerDied","Data":"847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9"}
Sep 08 07:21:10.328730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.328577    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"
Sep 08 07:21:10.328730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.328610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq" event={"ID":"01bdd106-6c50-40ff-9afb-f7b3e5d09ca6","Type":"ContainerDied","Data":"8999f8b2f6629df0a31e7c68b0257707c87df019463ed53f0a3eab3f8d90de1c"}
Sep 08 07:21:10.328730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.328632    2580 scope.go:117] "RemoveContainer" containerID="847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9"
Sep 08 07:21:10.337780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.337762    2580 scope.go:117] "RemoveContainer" containerID="847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9"
Sep 08 07:21:10.338053 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:21:10.338033    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9\": container with ID starting with 847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9 not found: ID does not exist" containerID="847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9"
Sep 08 07:21:10.338134 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.338061    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9"} err="failed to get container status \"847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9\": rpc error: code = NotFound desc = could not find container \"847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9\": container with ID starting with 847b438fda6aca128c581e9d934e1c1365b3a8e0c157a5a0eb83b17c1404d3d9 not found: ID does not exist"
Sep 08 07:21:10.349955 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.349932    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"]
Sep 08 07:21:10.353370 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.353344    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-da54544b-769df9c8f7-lr8sq"]
Sep 08 07:21:10.576791 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.576754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:10.885384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.885355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:10.940383 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.940343    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01bdd106-6c50-40ff-9afb-f7b3e5d09ca6" path="/var/lib/kubelet/pods/01bdd106-6c50-40ff-9afb-f7b3e5d09ca6/volumes"
Sep 08 07:21:10.987431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:10.987395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:12.072440 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:12.072406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:18.014563 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:18.014528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:18.121583 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:18.121544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:18.235228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:18.235196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:18.324527 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:18.324496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:18.656365 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:18.656286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:20.050365 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.050330    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"]
Sep 08 07:21:20.050750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.050720    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3befb5eb-9437-41c3-bb59-e5b3884c3df4" containerName="payload-processing"
Sep 08 07:21:20.050750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.050733    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3befb5eb-9437-41c3-bb59-e5b3884c3df4" containerName="payload-processing"
Sep 08 07:21:20.050750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.050746    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01bdd106-6c50-40ff-9afb-f7b3e5d09ca6" containerName="payload-processing"
Sep 08 07:21:20.050750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.050752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01bdd106-6c50-40ff-9afb-f7b3e5d09ca6" containerName="payload-processing"
Sep 08 07:21:20.050922 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.050828    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01bdd106-6c50-40ff-9afb-f7b3e5d09ca6" containerName="payload-processing"
Sep 08 07:21:20.050922 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.050842    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3befb5eb-9437-41c3-bb59-e5b3884c3df4" containerName="payload-processing"
Sep 08 07:21:20.053897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.053878    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.055810 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.055789    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-f2324022-kserve-self-signed-certs\""
Sep 08 07:21:20.063664 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.063641    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"]
Sep 08 07:21:20.072901 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.072874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:20.168981 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.168936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-tmp-dir\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.169192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.169020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-model-cache\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.169192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.169055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-home\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.169192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.169090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-dshm\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.169192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.169123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklmr\" (UniqueName: \"kubernetes.io/projected/a1ae8436-159d-47be-9f85-4c284c140e5b-kube-api-access-jklmr\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.169192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.169158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ae8436-159d-47be-9f85-4c284c140e5b-tls-certs\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270090 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-tmp-dir\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270313 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-model-cache\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270313 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-home\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270313 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-dshm\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270313 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklmr\" (UniqueName: \"kubernetes.io/projected/a1ae8436-159d-47be-9f85-4c284c140e5b-kube-api-access-jklmr\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270313 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ae8436-159d-47be-9f85-4c284c140e5b-tls-certs\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270621 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-tmp-dir\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270673 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-home\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.270709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.270693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-model-cache\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.272555 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.272533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-dshm\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.272712 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.272694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ae8436-159d-47be-9f85-4c284c140e5b-tls-certs\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.281347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.281318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklmr\" (UniqueName: \"kubernetes.io/projected/a1ae8436-159d-47be-9f85-4c284c140e5b-kube-api-access-jklmr\") pod \"e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") " pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.363724 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.363667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:20.640709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:20.640681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"]
Sep 08 07:21:20.643172 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:21:20.643140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1ae8436_159d_47be_9f85_4c284c140e5b.slice/crio-45492697a1ce27b61de893b56ec09af941cba2dc3f61b75f7ceeb812a14951ca WatchSource:0}: Error finding container 45492697a1ce27b61de893b56ec09af941cba2dc3f61b75f7ceeb812a14951ca: Status 404 returned error can't find the container with id 45492697a1ce27b61de893b56ec09af941cba2dc3f61b75f7ceeb812a14951ca
Sep 08 07:21:21.392210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:21.392173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk" event={"ID":"a1ae8436-159d-47be-9f85-4c284c140e5b","Type":"ContainerStarted","Data":"2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09"}
Sep 08 07:21:21.392621 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:21.392223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk" event={"ID":"a1ae8436-159d-47be-9f85-4c284c140e5b","Type":"ContainerStarted","Data":"45492697a1ce27b61de893b56ec09af941cba2dc3f61b75f7ceeb812a14951ca"}
Sep 08 07:21:21.408917 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:21.408860    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk" podStartSLOduration=1.408841581 podStartE2EDuration="1.408841581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:21:21.406394122 +0000 UTC m=+997.089854286" watchObservedRunningTime="2026-09-08 07:21:21.408841581 +0000 UTC m=+997.092301733"
Sep 08 07:21:22.986977 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:22.986934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:25.548848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:25.548804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:25.782076 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:25.782038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:26.183811 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:26.183726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:26.463503 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:26.463385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:26.650729 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:26.650691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:29.544798 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:29.544762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:29.646038 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:29.646001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:30.364213 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:30.364176    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:30.364413 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:30.364266    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:30.377095 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:30.377068    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:30.444494 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:30.444444    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:21:30.707661 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:30.707549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:30.807093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:30.807053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:31.020820 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.020703    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:31.375369 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.375321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:31.475594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.475554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:31.512622 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.512585    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"]
Sep 08 07:21:31.515968 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.515951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:31.517858 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.517836    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-48a836-dockercfg-h7d25\""
Sep 08 07:21:31.517980 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.517934    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-48a836\""
Sep 08 07:21:31.525113 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.525085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"]
Sep 08 07:21:31.682944 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.682851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521ab447-75bc-4cb8-a210-06625e37b883-plugins-config-volume\") pod \"payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw\" (UID: \"521ab447-75bc-4cb8-a210-06625e37b883\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:31.683119 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.682968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbfv\" (UniqueName: \"kubernetes.io/projected/521ab447-75bc-4cb8-a210-06625e37b883-kube-api-access-rpbfv\") pod \"payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw\" (UID: \"521ab447-75bc-4cb8-a210-06625e37b883\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:31.784136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.784087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521ab447-75bc-4cb8-a210-06625e37b883-plugins-config-volume\") pod \"payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw\" (UID: \"521ab447-75bc-4cb8-a210-06625e37b883\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:31.784590 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.784202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbfv\" (UniqueName: \"kubernetes.io/projected/521ab447-75bc-4cb8-a210-06625e37b883-kube-api-access-rpbfv\") pod \"payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw\" (UID: \"521ab447-75bc-4cb8-a210-06625e37b883\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:31.784841 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.784821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521ab447-75bc-4cb8-a210-06625e37b883-plugins-config-volume\") pod \"payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw\" (UID: \"521ab447-75bc-4cb8-a210-06625e37b883\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:31.792790 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.792762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbfv\" (UniqueName: \"kubernetes.io/projected/521ab447-75bc-4cb8-a210-06625e37b883-kube-api-access-rpbfv\") pod \"payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw\" (UID: \"521ab447-75bc-4cb8-a210-06625e37b883\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:31.827417 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:31.827387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:32.098662 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:32.098637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"]
Sep 08 07:21:32.100863 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:21:32.100831    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521ab447_75bc_4cb8_a210_06625e37b883.slice/crio-b98c46a663e1c94e730985804d13a1223c1514aa0aa529d9fb9da5b6522e5931 WatchSource:0}: Error finding container b98c46a663e1c94e730985804d13a1223c1514aa0aa529d9fb9da5b6522e5931: Status 404 returned error can't find the container with id b98c46a663e1c94e730985804d13a1223c1514aa0aa529d9fb9da5b6522e5931
Sep 08 07:21:32.445448 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:32.445357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw" event={"ID":"521ab447-75bc-4cb8-a210-06625e37b883","Type":"ContainerStarted","Data":"77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817"}
Sep 08 07:21:32.445448 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:32.445414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw" event={"ID":"521ab447-75bc-4cb8-a210-06625e37b883","Type":"ContainerStarted","Data":"b98c46a663e1c94e730985804d13a1223c1514aa0aa529d9fb9da5b6522e5931"}
Sep 08 07:21:32.462936 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:32.462881    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw" podStartSLOduration=1.4628618979999999 podStartE2EDuration="1.462861898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:21:32.459499405 +0000 UTC m=+1008.142959554" watchObservedRunningTime="2026-09-08 07:21:32.462861898 +0000 UTC m=+1008.146322055"
Sep 08 07:21:33.450070 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:33.450040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:44.457004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:44.456965    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:21:44.544583 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:44.544547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:50.542572 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:50.542506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:50.601435 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:50.601393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:50.679849 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:50.679798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:53.605500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:53.605455    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"]
Sep 08 07:21:53.705575 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:53.705534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:54.557130 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:54.557079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:55.631997 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:55.631958    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"]
Sep 08 07:21:55.644543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:55.644509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:21:59.271493 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:21:59.271426    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:06.522331 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:06.522291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:08.625368 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.625331    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk" podUID="a1ae8436-159d-47be-9f85-4c284c140e5b" containerName="main" containerID="cri-o://2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09" gracePeriod=45
Sep 08 07:22:08.877301 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.877231    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:22:08.943985 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.943954    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ae8436-159d-47be-9f85-4c284c140e5b-tls-certs\") pod \"a1ae8436-159d-47be-9f85-4c284c140e5b\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") "
Sep 08 07:22:08.944157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-tmp-dir\") pod \"a1ae8436-159d-47be-9f85-4c284c140e5b\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") "
Sep 08 07:22:08.944157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jklmr\" (UniqueName: \"kubernetes.io/projected/a1ae8436-159d-47be-9f85-4c284c140e5b-kube-api-access-jklmr\") pod \"a1ae8436-159d-47be-9f85-4c284c140e5b\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") "
Sep 08 07:22:08.944157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-dshm\") pod \"a1ae8436-159d-47be-9f85-4c284c140e5b\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") "
Sep 08 07:22:08.944334 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944230    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-model-cache\") pod \"a1ae8436-159d-47be-9f85-4c284c140e5b\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") "
Sep 08 07:22:08.944334 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944261    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-home\") pod \"a1ae8436-159d-47be-9f85-4c284c140e5b\" (UID: \"a1ae8436-159d-47be-9f85-4c284c140e5b\") "
Sep 08 07:22:08.944447 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944353    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1ae8436-159d-47be-9f85-4c284c140e5b" (UID: "a1ae8436-159d-47be-9f85-4c284c140e5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:22:08.944543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944498    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1ae8436-159d-47be-9f85-4c284c140e5b" (UID: "a1ae8436-159d-47be-9f85-4c284c140e5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:22:08.944543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944528    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-home" (OuterVolumeSpecName: "home") pod "a1ae8436-159d-47be-9f85-4c284c140e5b" (UID: "a1ae8436-159d-47be-9f85-4c284c140e5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:22:08.944656 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944642    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:08.944711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944660    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:08.944711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.944689    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:08.946308 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.946285    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-dshm" (OuterVolumeSpecName: "dshm") pod "a1ae8436-159d-47be-9f85-4c284c140e5b" (UID: "a1ae8436-159d-47be-9f85-4c284c140e5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:22:08.946412 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.946317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ae8436-159d-47be-9f85-4c284c140e5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1ae8436-159d-47be-9f85-4c284c140e5b" (UID: "a1ae8436-159d-47be-9f85-4c284c140e5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:22:08.946412 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:08.946324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ae8436-159d-47be-9f85-4c284c140e5b-kube-api-access-jklmr" (OuterVolumeSpecName: "kube-api-access-jklmr") pod "a1ae8436-159d-47be-9f85-4c284c140e5b" (UID: "a1ae8436-159d-47be-9f85-4c284c140e5b"). InnerVolumeSpecName "kube-api-access-jklmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:22:09.045592 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.045544    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ae8436-159d-47be-9f85-4c284c140e5b-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:09.045592 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.045582    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jklmr\" (UniqueName: \"kubernetes.io/projected/a1ae8436-159d-47be-9f85-4c284c140e5b-kube-api-access-jklmr\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:09.045592 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.045595    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ae8436-159d-47be-9f85-4c284c140e5b-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:09.628018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.625811    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1ae8436-159d-47be-9f85-4c284c140e5b" containerID="2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09" exitCode=0
Sep 08 07:22:09.628018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.625920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk" event={"ID":"a1ae8436-159d-47be-9f85-4c284c140e5b","Type":"ContainerDied","Data":"2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09"}
Sep 08 07:22:09.628018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.625970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk" event={"ID":"a1ae8436-159d-47be-9f85-4c284c140e5b","Type":"ContainerDied","Data":"45492697a1ce27b61de893b56ec09af941cba2dc3f61b75f7ceeb812a14951ca"}
Sep 08 07:22:09.628018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.625991    2580 scope.go:117] "RemoveContainer" containerID="2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09"
Sep 08 07:22:09.628018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.626203    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"
Sep 08 07:22:09.637202 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.637184    2580 scope.go:117] "RemoveContainer" containerID="2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09"
Sep 08 07:22:09.637492 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:22:09.637470    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09\": container with ID starting with 2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09 not found: ID does not exist" containerID="2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09"
Sep 08 07:22:09.637595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.637498    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09"} err="failed to get container status \"2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09\": rpc error: code = NotFound desc = could not find container \"2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09\": container with ID starting with 2b4d50eb54c2da73d92d474240f4b3cc1311062d56729e0a6d992b30de990b09 not found: ID does not exist"
Sep 08 07:22:09.647763 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.647740    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"]
Sep 08 07:22:09.651720 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:09.651699    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-f2324022-kserve-7c794b6f7b-msmgk"]
Sep 08 07:22:10.652394 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:10.652356    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd" podUID="22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" containerName="main" containerID="cri-o://4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636" gracePeriod=45
Sep 08 07:22:10.907004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:10.906946    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:22:10.939951 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:10.939918    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1ae8436-159d-47be-9f85-4c284c140e5b" path="/var/lib/kubelet/pods/a1ae8436-159d-47be-9f85-4c284c140e5b/volumes"
Sep 08 07:22:11.063590 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.063559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2rgx\" (UniqueName: \"kubernetes.io/projected/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-kube-api-access-q2rgx\") pod \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") "
Sep 08 07:22:11.063590 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.063602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tls-certs\") pod \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") "
Sep 08 07:22:11.063860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.063683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-model-cache\") pod \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") "
Sep 08 07:22:11.063860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.063753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-home\") pod \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") "
Sep 08 07:22:11.063860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.063786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-dshm\") pod \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") "
Sep 08 07:22:11.063860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.063836    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tmp-dir\") pod \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\" (UID: \"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c\") "
Sep 08 07:22:11.064053 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.063986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-model-cache" (OuterVolumeSpecName: "model-cache") pod "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" (UID: "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:22:11.064053 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.064006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-home" (OuterVolumeSpecName: "home") pod "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" (UID: "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:22:11.064171 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.064112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" (UID: "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:22:11.064259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.064227    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:11.064327 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.064260    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:11.064327 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.064269    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:11.065790 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.065763    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-kube-api-access-q2rgx" (OuterVolumeSpecName: "kube-api-access-q2rgx") pod "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" (UID: "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c"). InnerVolumeSpecName "kube-api-access-q2rgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:22:11.066211 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.066193    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" (UID: "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:22:11.066290 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.066206    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-dshm" (OuterVolumeSpecName: "dshm") pod "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" (UID: "22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:22:11.165515 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.165404    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:11.165515 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.165434    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2rgx\" (UniqueName: \"kubernetes.io/projected/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-kube-api-access-q2rgx\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:11.165515 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.165445    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:11.637729 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.637697    2580 generic.go:356] "Generic (PLEG): container finished" podID="22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" containerID="4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636" exitCode=0
Sep 08 07:22:11.637924 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.637743    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"
Sep 08 07:22:11.637924 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.637752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd" event={"ID":"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c","Type":"ContainerDied","Data":"4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636"}
Sep 08 07:22:11.637924 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.637783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd" event={"ID":"22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c","Type":"ContainerDied","Data":"96e369444050802453e25b9eae798ff5aa144f2690db52b6000dca4cd8aef8de"}
Sep 08 07:22:11.637924 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.637800    2580 scope.go:117] "RemoveContainer" containerID="4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636"
Sep 08 07:22:11.647245 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.647227    2580 scope.go:117] "RemoveContainer" containerID="4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636"
Sep 08 07:22:11.647542 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:22:11.647516    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636\": container with ID starting with 4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636 not found: ID does not exist" containerID="4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636"
Sep 08 07:22:11.647599 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.647551    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636"} err="failed to get container status \"4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636\": rpc error: code = NotFound desc = could not find container \"4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636\": container with ID starting with 4a5f7598bbbd7089de861019222d4532295049bc5dabe2c0e4804f6b532d9636 not found: ID does not exist"
Sep 08 07:22:11.658582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.658555    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"]
Sep 08 07:22:11.664418 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:11.664398    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-f2324022-kserve-b97d7754b-f9cwd"]
Sep 08 07:22:12.395195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:12.395151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:12.499387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:12.499353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:12.596283 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:12.596246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:12.940489 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:12.940445    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" path="/var/lib/kubelet/pods/22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c/volumes"
Sep 08 07:22:15.052591 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.052551    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"]
Sep 08 07:22:15.053009 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.052902    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ae8436-159d-47be-9f85-4c284c140e5b" containerName="main"
Sep 08 07:22:15.053009 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.052913    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ae8436-159d-47be-9f85-4c284c140e5b" containerName="main"
Sep 08 07:22:15.053009 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.052927    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" containerName="main"
Sep 08 07:22:15.053009 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.052932    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" containerName="main"
Sep 08 07:22:15.053009 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.052991    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ae8436-159d-47be-9f85-4c284c140e5b" containerName="main"
Sep 08 07:22:15.053009 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.053001    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d0e4d4-5e0f-4e9a-83d2-1ad9a8d1131c" containerName="main"
Sep 08 07:22:15.056104 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.056087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.058686 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.058664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6f5c4f\"/\"test-model-6f5c4f-kserve-self-signed-certs\""
Sep 08 07:22:15.058822 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.058664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6f5c4f\"/\"default-dockercfg-nlw5l\""
Sep 08 07:22:15.058822 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.058665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6f5c4f\"/\"openshift-service-ca.crt\""
Sep 08 07:22:15.058822 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.058731    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6f5c4f\"/\"kube-root-ca.crt\""
Sep 08 07:22:15.064787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.064759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"]
Sep 08 07:22:15.141384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.141353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:15.204256 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.204215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4wc\" (UniqueName: \"kubernetes.io/projected/18614518-b489-4104-8c8a-81cd74fd7cf0-kube-api-access-rv4wc\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.204256 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.204258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-model-cache\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.204518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.204360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-home\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.204518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.204388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-dshm\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.204518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.204485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18614518-b489-4104-8c8a-81cd74fd7cf0-tls-certs\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.204663 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.204552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-tmp-dir\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306088 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-home\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306088 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-dshm\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18614518-b489-4104-8c8a-81cd74fd7cf0-tls-certs\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-tmp-dir\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4wc\" (UniqueName: \"kubernetes.io/projected/18614518-b489-4104-8c8a-81cd74fd7cf0-kube-api-access-rv4wc\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-model-cache\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-home\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-tmp-dir\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.306614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.306564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-model-cache\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.308364 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.308333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-dshm\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.308735 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.308716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18614518-b489-4104-8c8a-81cd74fd7cf0-tls-certs\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.314479 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.314443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4wc\" (UniqueName: \"kubernetes.io/projected/18614518-b489-4104-8c8a-81cd74fd7cf0-kube-api-access-rv4wc\") pod \"test-model-6f5c4f-kserve-7dbbfd886f-6mq7w\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") " pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.366836 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.366791    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:15.638730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.638703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"]
Sep 08 07:22:15.640630 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:22:15.640600    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18614518_b489_4104_8c8a_81cd74fd7cf0.slice/crio-9b5780a2c17967dfa2c488d4808e8b602af32d39245fdca37e57b69130cd79e8 WatchSource:0}: Error finding container 9b5780a2c17967dfa2c488d4808e8b602af32d39245fdca37e57b69130cd79e8: Status 404 returned error can't find the container with id 9b5780a2c17967dfa2c488d4808e8b602af32d39245fdca37e57b69130cd79e8
Sep 08 07:22:15.659164 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:15.659134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w" event={"ID":"18614518-b489-4104-8c8a-81cd74fd7cf0","Type":"ContainerStarted","Data":"9b5780a2c17967dfa2c488d4808e8b602af32d39245fdca37e57b69130cd79e8"}
Sep 08 07:22:16.665452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:16.665419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w" event={"ID":"18614518-b489-4104-8c8a-81cd74fd7cf0","Type":"ContainerStarted","Data":"0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc"}
Sep 08 07:22:16.684244 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:16.684190    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w" podStartSLOduration=1.684171479 podStartE2EDuration="1.684171479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:22:16.682909328 +0000 UTC m=+1052.366369481" watchObservedRunningTime="2026-09-08 07:22:16.684171479 +0000 UTC m=+1052.367631632"
Sep 08 07:22:17.400551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:17.400509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:23.025912 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:23.025877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:23.739696 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:23.739661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:25.367187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:25.367145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:25.367616 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:25.367210    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:25.380032 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:25.380003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:25.718192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:25.718113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:22:27.993315 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:27.993283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:28.016649 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.016617    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"]
Sep 08 07:22:28.016853 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.016826    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw" podUID="521ab447-75bc-4cb8-a210-06625e37b883" containerName="payload-processing" containerID="cri-o://77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817" gracePeriod=30
Sep 08 07:22:28.093813 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.093781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:28.191631 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.191602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:28.245840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.245782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:28.253738 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.253715    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:22:28.332493 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.332443    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521ab447-75bc-4cb8-a210-06625e37b883-plugins-config-volume\") pod \"521ab447-75bc-4cb8-a210-06625e37b883\" (UID: \"521ab447-75bc-4cb8-a210-06625e37b883\") "
Sep 08 07:22:28.332637 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.332593    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpbfv\" (UniqueName: \"kubernetes.io/projected/521ab447-75bc-4cb8-a210-06625e37b883-kube-api-access-rpbfv\") pod \"521ab447-75bc-4cb8-a210-06625e37b883\" (UID: \"521ab447-75bc-4cb8-a210-06625e37b883\") "
Sep 08 07:22:28.332793 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.332771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521ab447-75bc-4cb8-a210-06625e37b883-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "521ab447-75bc-4cb8-a210-06625e37b883" (UID: "521ab447-75bc-4cb8-a210-06625e37b883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:22:28.334709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.334679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521ab447-75bc-4cb8-a210-06625e37b883-kube-api-access-rpbfv" (OuterVolumeSpecName: "kube-api-access-rpbfv") pod "521ab447-75bc-4cb8-a210-06625e37b883" (UID: "521ab447-75bc-4cb8-a210-06625e37b883"). InnerVolumeSpecName "kube-api-access-rpbfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:22:28.434120 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.434071    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbfv\" (UniqueName: \"kubernetes.io/projected/521ab447-75bc-4cb8-a210-06625e37b883-kube-api-access-rpbfv\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:28.434120 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.434112    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521ab447-75bc-4cb8-a210-06625e37b883-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:22:28.511372 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.511295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:28.723615 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.723582    2580 generic.go:356] "Generic (PLEG): container finished" podID="521ab447-75bc-4cb8-a210-06625e37b883" containerID="77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817" exitCode=0
Sep 08 07:22:28.723772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.723626    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"
Sep 08 07:22:28.723772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.723669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw" event={"ID":"521ab447-75bc-4cb8-a210-06625e37b883","Type":"ContainerDied","Data":"77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817"}
Sep 08 07:22:28.723772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.723722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw" event={"ID":"521ab447-75bc-4cb8-a210-06625e37b883","Type":"ContainerDied","Data":"b98c46a663e1c94e730985804d13a1223c1514aa0aa529d9fb9da5b6522e5931"}
Sep 08 07:22:28.723772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.723746    2580 scope.go:117] "RemoveContainer" containerID="77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817"
Sep 08 07:22:28.732569 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.732549    2580 scope.go:117] "RemoveContainer" containerID="77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817"
Sep 08 07:22:28.732816 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:22:28.732797    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817\": container with ID starting with 77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817 not found: ID does not exist" containerID="77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817"
Sep 08 07:22:28.732868 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.732823    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817"} err="failed to get container status \"77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817\": rpc error: code = NotFound desc = could not find container \"77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817\": container with ID starting with 77aa036692e0b08a9d35919a4e3d37e7017026fda4fa932a76b39af9c4ee4817 not found: ID does not exist"
Sep 08 07:22:28.745435 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.745384    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"]
Sep 08 07:22:28.747137 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.747112    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a836-6b956fdffc-xr5gw"]
Sep 08 07:22:28.940381 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:28.940352    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="521ab447-75bc-4cb8-a210-06625e37b883" path="/var/lib/kubelet/pods/521ab447-75bc-4cb8-a210-06625e37b883/volumes"
Sep 08 07:22:29.824703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:29.824669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:29.926514 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:29.926482    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:30.024862 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:30.024832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:30.125136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:30.125051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:30.665451 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:30.665413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:30.765612 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:30.765513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:30.817381 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:30.817332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:32.234675 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:32.234644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:34.493446 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:34.493407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:34.593683 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:34.593651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:34.733356 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:34.733305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:35.143605 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:35.143573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:35.426330 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:35.426211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:35.729522 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:35.729429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:36.345314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:36.345284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:36.445591 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:36.445541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:36.545390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:36.545360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:38.089870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:38.089773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:38.190451 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:38.190418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:38.764704 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:38.764662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:42.923826 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.923791    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"]
Sep 08 07:22:42.924212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.924182    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="521ab447-75bc-4cb8-a210-06625e37b883" containerName="payload-processing"
Sep 08 07:22:42.924212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.924194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="521ab447-75bc-4cb8-a210-06625e37b883" containerName="payload-processing"
Sep 08 07:22:42.924287 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.924279    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="521ab447-75bc-4cb8-a210-06625e37b883" containerName="payload-processing"
Sep 08 07:22:42.927291 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.927273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:42.929122 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.929101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6d29c3\"/\"noten-model-6d29c3-kserve-self-signed-certs\""
Sep 08 07:22:42.929772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.929752    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6d29c3\"/\"openshift-service-ca.crt\""
Sep 08 07:22:42.929874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.929807    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6d29c3\"/\"kube-root-ca.crt\""
Sep 08 07:22:42.929874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.929854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6d29c3\"/\"default-dockercfg-tl7qc\""
Sep 08 07:22:42.941091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:42.941068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"]
Sep 08 07:22:43.067801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.067764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-dshm\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.068024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.067810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368d6204-04dd-403a-9205-c1b32e634769-tls-certs\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.068024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.067832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-model-cache\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.068024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.067885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-845lw\" (UniqueName: \"kubernetes.io/projected/368d6204-04dd-403a-9205-c1b32e634769-kube-api-access-845lw\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.068024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.067935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-tmp-dir\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.068024 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.067984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-home\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.130630 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.130596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:43.169148 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-tmp-dir\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.169337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-home\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.169337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-dshm\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.169337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368d6204-04dd-403a-9205-c1b32e634769-tls-certs\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.169538 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-model-cache\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.169538 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-845lw\" (UniqueName: \"kubernetes.io/projected/368d6204-04dd-403a-9205-c1b32e634769-kube-api-access-845lw\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.169640 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-tmp-dir\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.169640 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-home\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.169848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.169819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-model-cache\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.171680 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.171653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-dshm\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.171774 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.171756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368d6204-04dd-403a-9205-c1b32e634769-tls-certs\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.177017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.176975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-845lw\" (UniqueName: \"kubernetes.io/projected/368d6204-04dd-403a-9205-c1b32e634769-kube-api-access-845lw\") pod \"noten-model-6d29c3-kserve-64c4b797d7-qb99n\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") " pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.238678 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.238638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:43.513578 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.513548    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"]
Sep 08 07:22:43.516397 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:22:43.516368    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod368d6204_04dd_403a_9205_c1b32e634769.slice/crio-8d8343d76403b59e7ac4d1c28e37288deaf849cfcb4b8173af9305554a2bf0b4 WatchSource:0}: Error finding container 8d8343d76403b59e7ac4d1c28e37288deaf849cfcb4b8173af9305554a2bf0b4: Status 404 returned error can't find the container with id 8d8343d76403b59e7ac4d1c28e37288deaf849cfcb4b8173af9305554a2bf0b4
Sep 08 07:22:43.797971 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.797884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n" event={"ID":"368d6204-04dd-403a-9205-c1b32e634769","Type":"ContainerStarted","Data":"0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d"}
Sep 08 07:22:43.797971 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.797925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n" event={"ID":"368d6204-04dd-403a-9205-c1b32e634769","Type":"ContainerStarted","Data":"8d8343d76403b59e7ac4d1c28e37288deaf849cfcb4b8173af9305554a2bf0b4"}
Sep 08 07:22:43.815573 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:43.815520    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n" podStartSLOduration=1.815498647 podStartE2EDuration="1.815498647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:22:43.813416262 +0000 UTC m=+1079.496876415" watchObservedRunningTime="2026-09-08 07:22:43.815498647 +0000 UTC m=+1079.498958801"
Sep 08 07:22:44.812695 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:44.812663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:44.872935 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:44.872899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:45.023085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:45.023039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:45.324510 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:45.324474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:45.713924 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:45.713839    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:48.020568 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:48.020531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:48.122318 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:48.122284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:49.170689 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:49.170656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:49.230501 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:49.230446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:49.395430 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:49.395399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:49.844745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:49.844715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:49.949828 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:49.949792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:22:53.238899 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:53.238860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:53.239299 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:53.238951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:53.251878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:53.251855    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:22:53.858899 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:22:53.858863    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:23:08.343394 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:08.343358    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"]
Sep 08 07:23:08.363334 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:08.363308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:08.481500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:08.481454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:08.701521 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:08.701428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:08.812503 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:08.812453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:12.906642 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:12.906606    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:13.518358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:13.518326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:13.623265 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:13.623231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:13.836493 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:13.836441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:13.934743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:13.934710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:18.122005 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:18.121951    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"]
Sep 08 07:23:18.143504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:18.143442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:19.190498 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.190436    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"]
Sep 08 07:23:19.218414 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.218379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:19.297362 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.297328    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"]
Sep 08 07:23:19.301124 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.301103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"
Sep 08 07:23:19.303182 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.303154    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:23:19.303339 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.303254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:23:19.307910 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.307882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"]
Sep 08 07:23:19.402488 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.402442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:19.418442 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.418403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6b1f34-e2de-4187-92cd-752266dfb095-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-620112-rkvds\" (UID: \"1d6b1f34-e2de-4187-92cd-752266dfb095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"
Sep 08 07:23:19.519482 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.519373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6b1f34-e2de-4187-92cd-752266dfb095-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-620112-rkvds\" (UID: \"1d6b1f34-e2de-4187-92cd-752266dfb095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"
Sep 08 07:23:19.520011 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.519989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6b1f34-e2de-4187-92cd-752266dfb095-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-620112-rkvds\" (UID: \"1d6b1f34-e2de-4187-92cd-752266dfb095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"
Sep 08 07:23:19.613351 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.613316    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"
Sep 08 07:23:19.882877 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.882851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"]
Sep 08 07:23:19.885482 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:23:19.885441    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6b1f34_e2de_4187_92cd_752266dfb095.slice/crio-727dbe57a878198035698595b5bb8635bbb031733b017fda35893adfcae929df WatchSource:0}: Error finding container 727dbe57a878198035698595b5bb8635bbb031733b017fda35893adfcae929df: Status 404 returned error can't find the container with id 727dbe57a878198035698595b5bb8635bbb031733b017fda35893adfcae929df
Sep 08 07:23:19.972528 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:19.972494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds" event={"ID":"1d6b1f34-e2de-4187-92cd-752266dfb095","Type":"ContainerStarted","Data":"727dbe57a878198035698595b5bb8635bbb031733b017fda35893adfcae929df"}
Sep 08 07:23:20.594200 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:20.594163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:20.648113 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:20.648073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:20.979757 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:20.979657    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d6b1f34-e2de-4187-92cd-752266dfb095" containerID="fbd6e135977c388b0e034f22154508118497229e26e3b2e66803f219ddf2c612" exitCode=0
Sep 08 07:23:20.979757 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:20.979726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds" event={"ID":"1d6b1f34-e2de-4187-92cd-752266dfb095","Type":"ContainerDied","Data":"fbd6e135977c388b0e034f22154508118497229e26e3b2e66803f219ddf2c612"}
Sep 08 07:23:21.068251 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:21.068193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:22.115840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:22.115815    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"
Sep 08 07:23:22.246274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:22.246189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6b1f34-e2de-4187-92cd-752266dfb095-maas-api-service-ca\") pod \"1d6b1f34-e2de-4187-92cd-752266dfb095\" (UID: \"1d6b1f34-e2de-4187-92cd-752266dfb095\") "
Sep 08 07:23:22.246580 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:22.246556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6b1f34-e2de-4187-92cd-752266dfb095-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d6b1f34-e2de-4187-92cd-752266dfb095" (UID: "1d6b1f34-e2de-4187-92cd-752266dfb095"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:23:22.347387 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:22.347351    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6b1f34-e2de-4187-92cd-752266dfb095-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:22.990244 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:22.990218    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"
Sep 08 07:23:22.990244 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:22.990240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds" event={"ID":"1d6b1f34-e2de-4187-92cd-752266dfb095","Type":"ContainerDied","Data":"727dbe57a878198035698595b5bb8635bbb031733b017fda35893adfcae929df"}
Sep 08 07:23:22.990435 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:22.990279    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="727dbe57a878198035698595b5bb8635bbb031733b017fda35893adfcae929df"
Sep 08 07:23:23.363113 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.363067    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n" podUID="368d6204-04dd-403a-9205-c1b32e634769" containerName="main" containerID="cri-o://0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d" gracePeriod=45
Sep 08 07:23:23.624422 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.624358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:23:23.760020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.759978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-home\") pod \"368d6204-04dd-403a-9205-c1b32e634769\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") "
Sep 08 07:23:23.760020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760025    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368d6204-04dd-403a-9205-c1b32e634769-tls-certs\") pod \"368d6204-04dd-403a-9205-c1b32e634769\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") "
Sep 08 07:23:23.760289 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-tmp-dir\") pod \"368d6204-04dd-403a-9205-c1b32e634769\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") "
Sep 08 07:23:23.760289 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-model-cache\") pod \"368d6204-04dd-403a-9205-c1b32e634769\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") "
Sep 08 07:23:23.760289 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-dshm\") pod \"368d6204-04dd-403a-9205-c1b32e634769\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") "
Sep 08 07:23:23.760289 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760167    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-845lw\" (UniqueName: \"kubernetes.io/projected/368d6204-04dd-403a-9205-c1b32e634769-kube-api-access-845lw\") pod \"368d6204-04dd-403a-9205-c1b32e634769\" (UID: \"368d6204-04dd-403a-9205-c1b32e634769\") "
Sep 08 07:23:23.760500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-home" (OuterVolumeSpecName: "home") pod "368d6204-04dd-403a-9205-c1b32e634769" (UID: "368d6204-04dd-403a-9205-c1b32e634769"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:23.760500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-model-cache" (OuterVolumeSpecName: "model-cache") pod "368d6204-04dd-403a-9205-c1b32e634769" (UID: "368d6204-04dd-403a-9205-c1b32e634769"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:23.760500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "368d6204-04dd-403a-9205-c1b32e634769" (UID: "368d6204-04dd-403a-9205-c1b32e634769"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:23.760668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760580    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:23.760668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760599    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:23.760668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.760613    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:23.762377 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.762350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/368d6204-04dd-403a-9205-c1b32e634769-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "368d6204-04dd-403a-9205-c1b32e634769" (UID: "368d6204-04dd-403a-9205-c1b32e634769"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:23:23.762490 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.762423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-dshm" (OuterVolumeSpecName: "dshm") pod "368d6204-04dd-403a-9205-c1b32e634769" (UID: "368d6204-04dd-403a-9205-c1b32e634769"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:23.762490 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.762436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368d6204-04dd-403a-9205-c1b32e634769-kube-api-access-845lw" (OuterVolumeSpecName: "kube-api-access-845lw") pod "368d6204-04dd-403a-9205-c1b32e634769" (UID: "368d6204-04dd-403a-9205-c1b32e634769"). InnerVolumeSpecName "kube-api-access-845lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:23:23.861625 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.861589    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368d6204-04dd-403a-9205-c1b32e634769-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:23.861625 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.861620    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368d6204-04dd-403a-9205-c1b32e634769-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:23.861625 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.861630    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-845lw\" (UniqueName: \"kubernetes.io/projected/368d6204-04dd-403a-9205-c1b32e634769-kube-api-access-845lw\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:23.996781 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.996692    2580 generic.go:356] "Generic (PLEG): container finished" podID="368d6204-04dd-403a-9205-c1b32e634769" containerID="0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d" exitCode=0
Sep 08 07:23:23.996781 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.996734    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"
Sep 08 07:23:23.996781 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.996758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n" event={"ID":"368d6204-04dd-403a-9205-c1b32e634769","Type":"ContainerDied","Data":"0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d"}
Sep 08 07:23:23.997003 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.996797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n" event={"ID":"368d6204-04dd-403a-9205-c1b32e634769","Type":"ContainerDied","Data":"8d8343d76403b59e7ac4d1c28e37288deaf849cfcb4b8173af9305554a2bf0b4"}
Sep 08 07:23:23.997003 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:23.996817    2580 scope.go:117] "RemoveContainer" containerID="0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d"
Sep 08 07:23:24.005727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:24.005708    2580 scope.go:117] "RemoveContainer" containerID="0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d"
Sep 08 07:23:24.005999 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:23:24.005982    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d\": container with ID starting with 0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d not found: ID does not exist" containerID="0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d"
Sep 08 07:23:24.006048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:24.006007    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d"} err="failed to get container status \"0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d\": rpc error: code = NotFound desc = could not find container \"0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d\": container with ID starting with 0a6a62df170a645cde065327683ca42047a46433a8f83ac6338d349256d7028d not found: ID does not exist"
Sep 08 07:23:24.017478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:24.017441    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"]
Sep 08 07:23:24.021689 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:24.021668    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6d29c3/noten-model-6d29c3-kserve-64c4b797d7-qb99n"]
Sep 08 07:23:24.941691 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:24.941655    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="368d6204-04dd-403a-9205-c1b32e634769" path="/var/lib/kubelet/pods/368d6204-04dd-403a-9205-c1b32e634769/volumes"
Sep 08 07:23:29.447922 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:29.447887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:30.029256 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:30.029227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:30.096195 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:30.096160    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:30.196866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:30.196834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:30.224785 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:30.224752    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"]
Sep 08 07:23:30.227334 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:30.227312    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-620112-rkvds"]
Sep 08 07:23:30.259428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:30.259398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:30.574729 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:30.574690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:30.941498 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:30.941445    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d6b1f34-e2de-4187-92cd-752266dfb095" path="/var/lib/kubelet/pods/1d6b1f34-e2de-4187-92cd-752266dfb095/volumes"
Sep 08 07:23:31.068403 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:31.068371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:31.656742 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:31.656711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:31.732524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:31.732490    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:31.835524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:31.835489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:31.902570 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:31.902538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:32.203384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:32.203349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:33.142934 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.142891    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8" podUID="13c92dec-10c8-405c-8242-d8e80cffe6ce" containerName="main" containerID="cri-o://f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae" gracePeriod=45
Sep 08 07:23:33.394563 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.394501    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:23:33.554912 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.554877    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9vgp\" (UniqueName: \"kubernetes.io/projected/13c92dec-10c8-405c-8242-d8e80cffe6ce-kube-api-access-r9vgp\") pod \"13c92dec-10c8-405c-8242-d8e80cffe6ce\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") "
Sep 08 07:23:33.555061 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.554926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-tmp-dir\") pod \"13c92dec-10c8-405c-8242-d8e80cffe6ce\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") "
Sep 08 07:23:33.555061 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.554973    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-model-cache\") pod \"13c92dec-10c8-405c-8242-d8e80cffe6ce\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") "
Sep 08 07:23:33.555183 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c92dec-10c8-405c-8242-d8e80cffe6ce-tls-certs\") pod \"13c92dec-10c8-405c-8242-d8e80cffe6ce\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") "
Sep 08 07:23:33.555183 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555093    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-home\") pod \"13c92dec-10c8-405c-8242-d8e80cffe6ce\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") "
Sep 08 07:23:33.555183 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-dshm\") pod \"13c92dec-10c8-405c-8242-d8e80cffe6ce\" (UID: \"13c92dec-10c8-405c-8242-d8e80cffe6ce\") "
Sep 08 07:23:33.555340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555179    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "13c92dec-10c8-405c-8242-d8e80cffe6ce" (UID: "13c92dec-10c8-405c-8242-d8e80cffe6ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:33.555340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555219    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13c92dec-10c8-405c-8242-d8e80cffe6ce" (UID: "13c92dec-10c8-405c-8242-d8e80cffe6ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:33.555449 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-home" (OuterVolumeSpecName: "home") pod "13c92dec-10c8-405c-8242-d8e80cffe6ce" (UID: "13c92dec-10c8-405c-8242-d8e80cffe6ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:33.555567 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555451    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:33.555567 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555487    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:33.555567 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.555511    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:33.557215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.557191    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-dshm" (OuterVolumeSpecName: "dshm") pod "13c92dec-10c8-405c-8242-d8e80cffe6ce" (UID: "13c92dec-10c8-405c-8242-d8e80cffe6ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:33.557296 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.557235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c92dec-10c8-405c-8242-d8e80cffe6ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13c92dec-10c8-405c-8242-d8e80cffe6ce" (UID: "13c92dec-10c8-405c-8242-d8e80cffe6ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:23:33.557339 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.557288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c92dec-10c8-405c-8242-d8e80cffe6ce-kube-api-access-r9vgp" (OuterVolumeSpecName: "kube-api-access-r9vgp") pod "13c92dec-10c8-405c-8242-d8e80cffe6ce" (UID: "13c92dec-10c8-405c-8242-d8e80cffe6ce"). InnerVolumeSpecName "kube-api-access-r9vgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:23:33.656509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.656421    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9vgp\" (UniqueName: \"kubernetes.io/projected/13c92dec-10c8-405c-8242-d8e80cffe6ce-kube-api-access-r9vgp\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:33.656509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.656448    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c92dec-10c8-405c-8242-d8e80cffe6ce-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:33.656509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:33.656479    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c92dec-10c8-405c-8242-d8e80cffe6ce-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:34.054518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.054414    2580 generic.go:356] "Generic (PLEG): container finished" podID="13c92dec-10c8-405c-8242-d8e80cffe6ce" containerID="f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae" exitCode=0
Sep 08 07:23:34.054518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.054453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8" event={"ID":"13c92dec-10c8-405c-8242-d8e80cffe6ce","Type":"ContainerDied","Data":"f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae"}
Sep 08 07:23:34.054518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.054477    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"
Sep 08 07:23:34.054518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.054507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8" event={"ID":"13c92dec-10c8-405c-8242-d8e80cffe6ce","Type":"ContainerDied","Data":"7012057756eb3142f0b0b31918f97f5780ddfebdc61223bd3eaca61127be26bf"}
Sep 08 07:23:34.054841 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.054526    2580 scope.go:117] "RemoveContainer" containerID="f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae"
Sep 08 07:23:34.063893 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.063876    2580 scope.go:117] "RemoveContainer" containerID="f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae"
Sep 08 07:23:34.064171 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:23:34.064152    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae\": container with ID starting with f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae not found: ID does not exist" containerID="f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae"
Sep 08 07:23:34.064213 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.064180    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae"} err="failed to get container status \"f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae\": rpc error: code = NotFound desc = could not find container \"f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae\": container with ID starting with f877606551c2343daee6b769462ddcf42c9e9236637eb7769922c9ac983680ae not found: ID does not exist"
Sep 08 07:23:34.075244 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.075221    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"]
Sep 08 07:23:34.087045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.084727    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-620112/test-model-620112-kserve-86b89cb6b9-7tlm8"]
Sep 08 07:23:34.210229 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.210188    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w" podUID="18614518-b489-4104-8c8a-81cd74fd7cf0" containerName="main" containerID="cri-o://0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc" gracePeriod=45
Sep 08 07:23:34.454342 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.454312    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:23:34.565384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv4wc\" (UniqueName: \"kubernetes.io/projected/18614518-b489-4104-8c8a-81cd74fd7cf0-kube-api-access-rv4wc\") pod \"18614518-b489-4104-8c8a-81cd74fd7cf0\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") "
Sep 08 07:23:34.565384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-dshm\") pod \"18614518-b489-4104-8c8a-81cd74fd7cf0\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") "
Sep 08 07:23:34.565675 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-model-cache\") pod \"18614518-b489-4104-8c8a-81cd74fd7cf0\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") "
Sep 08 07:23:34.565675 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-home\") pod \"18614518-b489-4104-8c8a-81cd74fd7cf0\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") "
Sep 08 07:23:34.565675 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18614518-b489-4104-8c8a-81cd74fd7cf0-tls-certs\") pod \"18614518-b489-4104-8c8a-81cd74fd7cf0\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") "
Sep 08 07:23:34.565675 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-tmp-dir\") pod \"18614518-b489-4104-8c8a-81cd74fd7cf0\" (UID: \"18614518-b489-4104-8c8a-81cd74fd7cf0\") "
Sep 08 07:23:34.565866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565717    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-home" (OuterVolumeSpecName: "home") pod "18614518-b489-4104-8c8a-81cd74fd7cf0" (UID: "18614518-b489-4104-8c8a-81cd74fd7cf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:34.565866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "18614518-b489-4104-8c8a-81cd74fd7cf0" (UID: "18614518-b489-4104-8c8a-81cd74fd7cf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:34.565866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.565845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18614518-b489-4104-8c8a-81cd74fd7cf0" (UID: "18614518-b489-4104-8c8a-81cd74fd7cf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:34.567764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.567736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-dshm" (OuterVolumeSpecName: "dshm") pod "18614518-b489-4104-8c8a-81cd74fd7cf0" (UID: "18614518-b489-4104-8c8a-81cd74fd7cf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:23:34.567764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.567747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18614518-b489-4104-8c8a-81cd74fd7cf0-kube-api-access-rv4wc" (OuterVolumeSpecName: "kube-api-access-rv4wc") pod "18614518-b489-4104-8c8a-81cd74fd7cf0" (UID: "18614518-b489-4104-8c8a-81cd74fd7cf0"). InnerVolumeSpecName "kube-api-access-rv4wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:23:34.567865 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.567822    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18614518-b489-4104-8c8a-81cd74fd7cf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18614518-b489-4104-8c8a-81cd74fd7cf0" (UID: "18614518-b489-4104-8c8a-81cd74fd7cf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:23:34.666309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.666220    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv4wc\" (UniqueName: \"kubernetes.io/projected/18614518-b489-4104-8c8a-81cd74fd7cf0-kube-api-access-rv4wc\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:34.666309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.666252    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:34.666309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.666263    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:34.666309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.666272    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:34.666309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.666279    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18614518-b489-4104-8c8a-81cd74fd7cf0-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:34.666309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.666307    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18614518-b489-4104-8c8a-81cd74fd7cf0-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:34.941666 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:34.941582    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c92dec-10c8-405c-8242-d8e80cffe6ce" path="/var/lib/kubelet/pods/13c92dec-10c8-405c-8242-d8e80cffe6ce/volumes"
Sep 08 07:23:35.061046 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.061008    2580 generic.go:356] "Generic (PLEG): container finished" podID="18614518-b489-4104-8c8a-81cd74fd7cf0" containerID="0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc" exitCode=0
Sep 08 07:23:35.061192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.061087    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"
Sep 08 07:23:35.061192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.061107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w" event={"ID":"18614518-b489-4104-8c8a-81cd74fd7cf0","Type":"ContainerDied","Data":"0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc"}
Sep 08 07:23:35.061192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.061146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w" event={"ID":"18614518-b489-4104-8c8a-81cd74fd7cf0","Type":"ContainerDied","Data":"9b5780a2c17967dfa2c488d4808e8b602af32d39245fdca37e57b69130cd79e8"}
Sep 08 07:23:35.061192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.061168    2580 scope.go:117] "RemoveContainer" containerID="0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc"
Sep 08 07:23:35.069545 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.069522    2580 scope.go:117] "RemoveContainer" containerID="0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc"
Sep 08 07:23:35.069834 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:23:35.069816    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc\": container with ID starting with 0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc not found: ID does not exist" containerID="0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc"
Sep 08 07:23:35.069892 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.069850    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc"} err="failed to get container status \"0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc\": rpc error: code = NotFound desc = could not find container \"0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc\": container with ID starting with 0219c07944f88b61f97f1ed122ccae987f69f3c1eeafdd1c492c6c4aefb2befc not found: ID does not exist"
Sep 08 07:23:35.087470 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.087433    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"]
Sep 08 07:23:35.089707 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:35.089685    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6f5c4f/test-model-6f5c4f-kserve-7dbbfd886f-6mq7w"]
Sep 08 07:23:36.941324 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:36.941292    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18614518-b489-4104-8c8a-81cd74fd7cf0" path="/var/lib/kubelet/pods/18614518-b489-4104-8c8a-81cd74fd7cf0/volumes"
Sep 08 07:23:37.092545 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:37.092508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:37.157637 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:37.157602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:37.345763 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:37.345727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:37.874088 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:37.874057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:39.066737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:39.066699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:39.143365 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:39.143330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:39.311707 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:39.311674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:39.594210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:39.594176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:39.693910 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:39.693876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:39.843567 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:39.843524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:45.743609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:45.743577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:45.810433 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:45.810393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:45.891101 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:45.891064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:46.082637 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:46.082602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:48.233333 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.233299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:48.252898 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.252861    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"]
Sep 08 07:23:48.253137 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.253092    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc" podUID="b1e84e93-1464-4401-b3d4-a3d2da78bfca" containerName="payload-processing" containerID="cri-o://3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9" gracePeriod=30
Sep 08 07:23:48.261679 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.261654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"]
Sep 08 07:23:48.261876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.261849    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4" podUID="870d23ee-eb59-41ed-86dd-a24eabc72eed" containerName="payload-pre-processing" containerID="cri-o://9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35" gracePeriod=30
Sep 08 07:23:48.338158 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.338123    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:48.437076 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.437047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:48.521419 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.521389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:48.528349 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.528328    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:23:48.531063 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.531041    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:23:48.585750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.585721    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkqd9\" (UniqueName: \"kubernetes.io/projected/b1e84e93-1464-4401-b3d4-a3d2da78bfca-kube-api-access-vkqd9\") pod \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\" (UID: \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\") "
Sep 08 07:23:48.585750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.585753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1e84e93-1464-4401-b3d4-a3d2da78bfca-plugins-config-volume\") pod \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\" (UID: \"b1e84e93-1464-4401-b3d4-a3d2da78bfca\") "
Sep 08 07:23:48.585960 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.585785    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870d23ee-eb59-41ed-86dd-a24eabc72eed-plugins-config-volume\") pod \"870d23ee-eb59-41ed-86dd-a24eabc72eed\" (UID: \"870d23ee-eb59-41ed-86dd-a24eabc72eed\") "
Sep 08 07:23:48.585960 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.585804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsllg\" (UniqueName: \"kubernetes.io/projected/870d23ee-eb59-41ed-86dd-a24eabc72eed-kube-api-access-nsllg\") pod \"870d23ee-eb59-41ed-86dd-a24eabc72eed\" (UID: \"870d23ee-eb59-41ed-86dd-a24eabc72eed\") "
Sep 08 07:23:48.586127 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.586101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e84e93-1464-4401-b3d4-a3d2da78bfca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1e84e93-1464-4401-b3d4-a3d2da78bfca" (UID: "b1e84e93-1464-4401-b3d4-a3d2da78bfca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:23:48.586177 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.586156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870d23ee-eb59-41ed-86dd-a24eabc72eed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "870d23ee-eb59-41ed-86dd-a24eabc72eed" (UID: "870d23ee-eb59-41ed-86dd-a24eabc72eed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:23:48.587878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.587848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870d23ee-eb59-41ed-86dd-a24eabc72eed-kube-api-access-nsllg" (OuterVolumeSpecName: "kube-api-access-nsllg") pod "870d23ee-eb59-41ed-86dd-a24eabc72eed" (UID: "870d23ee-eb59-41ed-86dd-a24eabc72eed"). InnerVolumeSpecName "kube-api-access-nsllg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:23:48.587976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.587879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e84e93-1464-4401-b3d4-a3d2da78bfca-kube-api-access-vkqd9" (OuterVolumeSpecName: "kube-api-access-vkqd9") pod "b1e84e93-1464-4401-b3d4-a3d2da78bfca" (UID: "b1e84e93-1464-4401-b3d4-a3d2da78bfca"). InnerVolumeSpecName "kube-api-access-vkqd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:23:48.686470 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.686438    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkqd9\" (UniqueName: \"kubernetes.io/projected/b1e84e93-1464-4401-b3d4-a3d2da78bfca-kube-api-access-vkqd9\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:48.686601 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.686484    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1e84e93-1464-4401-b3d4-a3d2da78bfca-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:48.686601 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.686494    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870d23ee-eb59-41ed-86dd-a24eabc72eed-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:48.686601 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.686502    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsllg\" (UniqueName: \"kubernetes.io/projected/870d23ee-eb59-41ed-86dd-a24eabc72eed-kube-api-access-nsllg\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:23:48.779845 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:48.779773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:49.131344 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.131309    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1e84e93-1464-4401-b3d4-a3d2da78bfca" containerID="3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9" exitCode=0
Sep 08 07:23:49.131344 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.131345    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"
Sep 08 07:23:49.131594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.131396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc" event={"ID":"b1e84e93-1464-4401-b3d4-a3d2da78bfca","Type":"ContainerDied","Data":"3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9"}
Sep 08 07:23:49.131594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.131448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc" event={"ID":"b1e84e93-1464-4401-b3d4-a3d2da78bfca","Type":"ContainerDied","Data":"3598074156d7cb80574d71d63400dd8bdfabad822cffd4b6b6774aa946a4727f"}
Sep 08 07:23:49.131594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.131499    2580 scope.go:117] "RemoveContainer" containerID="3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9"
Sep 08 07:23:49.132885 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.132862    2580 generic.go:356] "Generic (PLEG): container finished" podID="870d23ee-eb59-41ed-86dd-a24eabc72eed" containerID="9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35" exitCode=0
Sep 08 07:23:49.133010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.132904    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"
Sep 08 07:23:49.133010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.132918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4" event={"ID":"870d23ee-eb59-41ed-86dd-a24eabc72eed","Type":"ContainerDied","Data":"9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35"}
Sep 08 07:23:49.133010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.132954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4" event={"ID":"870d23ee-eb59-41ed-86dd-a24eabc72eed","Type":"ContainerDied","Data":"b08e200ff32033e1c09a6bfb5ef7242ebeb97ab9ef2ed568d7aacdef03acc8f7"}
Sep 08 07:23:49.142268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.142241    2580 scope.go:117] "RemoveContainer" containerID="3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9"
Sep 08 07:23:49.142688 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:23:49.142667    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9\": container with ID starting with 3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9 not found: ID does not exist" containerID="3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9"
Sep 08 07:23:49.142780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.142700    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9"} err="failed to get container status \"3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9\": rpc error: code = NotFound desc = could not find container \"3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9\": container with ID starting with 3496a7ed4cea5dd80f16b8b5e6ed7b20a66bc53784482942c5b896a2d79805a9 not found: ID does not exist"
Sep 08 07:23:49.142780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.142725    2580 scope.go:117] "RemoveContainer" containerID="9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35"
Sep 08 07:23:49.150592 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.150572    2580 scope.go:117] "RemoveContainer" containerID="9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35"
Sep 08 07:23:49.150876 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:23:49.150855    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35\": container with ID starting with 9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35 not found: ID does not exist" containerID="9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35"
Sep 08 07:23:49.150926 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.150883    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35"} err="failed to get container status \"9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35\": rpc error: code = NotFound desc = could not find container \"9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35\": container with ID starting with 9c8c8a1d034e0cc616f5300178dde6d6a1e0af466ade3b17113839a07243cf35 not found: ID does not exist"
Sep 08 07:23:49.151744 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.151725    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"]
Sep 08 07:23:49.155570 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.155549    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f5c4f-67476678fd-kjqf4"]
Sep 08 07:23:49.165349 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.165328    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"]
Sep 08 07:23:49.167621 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:49.167601    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f5c4f-75bf968c6c-gvqpc"]
Sep 08 07:23:50.940366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:50.940333    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="870d23ee-eb59-41ed-86dd-a24eabc72eed" path="/var/lib/kubelet/pods/870d23ee-eb59-41ed-86dd-a24eabc72eed/volumes"
Sep 08 07:23:50.940745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:50.940697    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e84e93-1464-4401-b3d4-a3d2da78bfca" path="/var/lib/kubelet/pods/b1e84e93-1464-4401-b3d4-a3d2da78bfca/volumes"
Sep 08 07:23:50.992439 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:50.992413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:52.705265 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:52.705234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:52.762167 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:52.762138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:53.584551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:53.584523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:53.875511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:53.875403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:54.149266 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:54.149182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:55.323648 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:55.323607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:55.424669 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:55.424630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:55.567317 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:55.567278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:56.072299 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:56.072269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:56.361074 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:56.360986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:56.457856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:56.457821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:56.556788 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:56.556745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:57.616710 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:57.616658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:57.716889 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:57.716855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:57.977886 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:57.977800    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:58.094102 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.094061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:58.189626 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.189590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:58.196646 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.196616    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"]
Sep 08 07:23:58.196983 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.196970    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="870d23ee-eb59-41ed-86dd-a24eabc72eed" containerName="payload-pre-processing"
Sep 08 07:23:58.197026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.196985    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="870d23ee-eb59-41ed-86dd-a24eabc72eed" containerName="payload-pre-processing"
Sep 08 07:23:58.197026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.196993    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c92dec-10c8-405c-8242-d8e80cffe6ce" containerName="main"
Sep 08 07:23:58.197026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.196999    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c92dec-10c8-405c-8242-d8e80cffe6ce" containerName="main"
Sep 08 07:23:58.197026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197005    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6b1f34-e2de-4187-92cd-752266dfb095" containerName="revoke-keys"
Sep 08 07:23:58.197026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6b1f34-e2de-4187-92cd-752266dfb095" containerName="revoke-keys"
Sep 08 07:23:58.197026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197023    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="368d6204-04dd-403a-9205-c1b32e634769" containerName="main"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197030    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="368d6204-04dd-403a-9205-c1b32e634769" containerName="main"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197041    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e84e93-1464-4401-b3d4-a3d2da78bfca" containerName="payload-processing"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197049    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e84e93-1464-4401-b3d4-a3d2da78bfca" containerName="payload-processing"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197055    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18614518-b489-4104-8c8a-81cd74fd7cf0" containerName="main"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197060    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18614518-b489-4104-8c8a-81cd74fd7cf0" containerName="main"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197113    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c92dec-10c8-405c-8242-d8e80cffe6ce" containerName="main"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197119    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18614518-b489-4104-8c8a-81cd74fd7cf0" containerName="main"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197127    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="368d6204-04dd-403a-9205-c1b32e634769" containerName="main"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197133    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="870d23ee-eb59-41ed-86dd-a24eabc72eed" containerName="payload-pre-processing"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197140    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6b1f34-e2de-4187-92cd-752266dfb095" containerName="revoke-keys"
Sep 08 07:23:58.197207 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.197147    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e84e93-1464-4401-b3d4-a3d2da78bfca" containerName="payload-processing"
Sep 08 07:23:58.201602 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.201581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:58.203511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.203483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-37abc6\""
Sep 08 07:23:58.203643 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.203608    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-37abc6-dockercfg-n2vnr\""
Sep 08 07:23:58.210773 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.210745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"]
Sep 08 07:23:58.274645 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.274553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlf42\" (UniqueName: \"kubernetes.io/projected/6379ba01-5467-4d49-ab81-d3b828e9d639-kube-api-access-hlf42\") pod \"payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd\" (UID: \"6379ba01-5467-4d49-ab81-d3b828e9d639\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:58.274645 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.274606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6379ba01-5467-4d49-ab81-d3b828e9d639-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd\" (UID: \"6379ba01-5467-4d49-ab81-d3b828e9d639\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:58.375659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.375621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlf42\" (UniqueName: \"kubernetes.io/projected/6379ba01-5467-4d49-ab81-d3b828e9d639-kube-api-access-hlf42\") pod \"payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd\" (UID: \"6379ba01-5467-4d49-ab81-d3b828e9d639\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:58.375860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.375669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6379ba01-5467-4d49-ab81-d3b828e9d639-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd\" (UID: \"6379ba01-5467-4d49-ab81-d3b828e9d639\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:58.376310 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.376289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6379ba01-5467-4d49-ab81-d3b828e9d639-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd\" (UID: \"6379ba01-5467-4d49-ab81-d3b828e9d639\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:58.383931 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.383909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlf42\" (UniqueName: \"kubernetes.io/projected/6379ba01-5467-4d49-ab81-d3b828e9d639-kube-api-access-hlf42\") pod \"payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd\" (UID: \"6379ba01-5467-4d49-ab81-d3b828e9d639\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:58.512232 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.512190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:58.786170 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:58.786131    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"]
Sep 08 07:23:58.788890 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:23:58.788861    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6379ba01_5467_4d49_ab81_d3b828e9d639.slice/crio-6743acef3b25622b8691a8ff3f691a522a4af61ed4a1b2b2ff0b2c1bba711723 WatchSource:0}: Error finding container 6743acef3b25622b8691a8ff3f691a522a4af61ed4a1b2b2ff0b2c1bba711723: Status 404 returned error can't find the container with id 6743acef3b25622b8691a8ff3f691a522a4af61ed4a1b2b2ff0b2c1bba711723
Sep 08 07:23:59.183277 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:59.183188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd" event={"ID":"6379ba01-5467-4d49-ab81-d3b828e9d639","Type":"ContainerStarted","Data":"fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c"}
Sep 08 07:23:59.183277 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:59.183245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd" event={"ID":"6379ba01-5467-4d49-ab81-d3b828e9d639","Type":"ContainerStarted","Data":"6743acef3b25622b8691a8ff3f691a522a4af61ed4a1b2b2ff0b2c1bba711723"}
Sep 08 07:23:59.183482 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:59.183330    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:23:59.201669 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:59.201620    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd" podStartSLOduration=1.201603873 podStartE2EDuration="1.201603873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:23:59.199559487 +0000 UTC m=+1154.883019639" watchObservedRunningTime="2026-09-08 07:23:59.201603873 +0000 UTC m=+1154.885064026"
Sep 08 07:23:59.430522 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:59.430488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:23:59.528659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:23:59.528574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:00.592431 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:00.592404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:00.647277 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:00.647247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:00.847453 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:00.847365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:01.096617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.096574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:01.161473 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.161380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:01.266091 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.266055    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"]
Sep 08 07:24:01.269347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.269331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:01.271419 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.271399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-fb59c2-dockercfg-dng46\""
Sep 08 07:24:01.271569 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.271441    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-fb59c2\""
Sep 08 07:24:01.279861 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.279835    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"]
Sep 08 07:24:01.301922 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.301890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6b61d-77c8-4610-8022-531bbe7ffd63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h\" (UID: \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:01.302049 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.301950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sb8k\" (UniqueName: \"kubernetes.io/projected/cdf6b61d-77c8-4610-8022-531bbe7ffd63-kube-api-access-8sb8k\") pod \"payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h\" (UID: \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:01.402883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.402845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6b61d-77c8-4610-8022-531bbe7ffd63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h\" (UID: \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:01.403070 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.402910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sb8k\" (UniqueName: \"kubernetes.io/projected/cdf6b61d-77c8-4610-8022-531bbe7ffd63-kube-api-access-8sb8k\") pod \"payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h\" (UID: \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:01.403556 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.403534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6b61d-77c8-4610-8022-531bbe7ffd63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h\" (UID: \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:01.414184 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.414122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sb8k\" (UniqueName: \"kubernetes.io/projected/cdf6b61d-77c8-4610-8022-531bbe7ffd63-kube-api-access-8sb8k\") pod \"payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h\" (UID: \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:01.579932 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.579903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:01.848878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:01.848851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"]
Sep 08 07:24:01.851137 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:24:01.851109    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf6b61d_77c8_4610_8022_531bbe7ffd63.slice/crio-6d744e4d67440272a312e4434f5224592c288c75ebaee363d2418f0a028c71a1 WatchSource:0}: Error finding container 6d744e4d67440272a312e4434f5224592c288c75ebaee363d2418f0a028c71a1: Status 404 returned error can't find the container with id 6d744e4d67440272a312e4434f5224592c288c75ebaee363d2418f0a028c71a1
Sep 08 07:24:02.198735 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:02.198646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h" event={"ID":"cdf6b61d-77c8-4610-8022-531bbe7ffd63","Type":"ContainerStarted","Data":"5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1"}
Sep 08 07:24:02.198735 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:02.198698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h" event={"ID":"cdf6b61d-77c8-4610-8022-531bbe7ffd63","Type":"ContainerStarted","Data":"6d744e4d67440272a312e4434f5224592c288c75ebaee363d2418f0a028c71a1"}
Sep 08 07:24:02.198952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:02.198765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:02.213707 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:02.213652    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h" podStartSLOduration=1.213634794 podStartE2EDuration="1.213634794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:24:02.212830593 +0000 UTC m=+1157.896290747" watchObservedRunningTime="2026-09-08 07:24:02.213634794 +0000 UTC m=+1157.897094978"
Sep 08 07:24:10.190363 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:10.190283    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:24:13.206245 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:13.206204    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:24:19.650149 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:19.650111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:21.073756 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:21.073720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:21.330559 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:21.330482    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:21.576794 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:21.576763    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:21.672190 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:21.672100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:21.929753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:21.929670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:22.029331 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:22.029302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:24.352606 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:24.352574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:24.653285 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:24.653193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:24.957707 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:24.957629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:25.057366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:25.057334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:25.156876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:25.156848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:25.425567 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:25.425521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:26.173259 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.173228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:26.274017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.273981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:26.541870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.541786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:26.670479 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.670420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:26.760245 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.760206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:26.761014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.760992    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"]
Sep 08 07:24:26.764428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.764409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:26.766340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.766323    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-0ca807-dockercfg-cxc6n\""
Sep 08 07:24:26.766539 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.766521    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-0ca807\""
Sep 08 07:24:26.776441 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.776414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"]
Sep 08 07:24:26.800210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.800139    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"]
Sep 08 07:24:26.803488 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.803453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:26.815625 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.815601    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"]
Sep 08 07:24:26.843312 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.843283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dbbc29-ac14-4cf6-86ef-03c317976256-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc\" (UID: \"41dbbc29-ac14-4cf6-86ef-03c317976256\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:26.843487 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.843327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b215da-5dde-4725-b707-18b8aa4737ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g\" (UID: \"04b215da-5dde-4725-b707-18b8aa4737ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:26.843487 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.843344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvfc\" (UniqueName: \"kubernetes.io/projected/41dbbc29-ac14-4cf6-86ef-03c317976256-kube-api-access-bsvfc\") pod \"payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc\" (UID: \"41dbbc29-ac14-4cf6-86ef-03c317976256\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:26.843619 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.843488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrm5\" (UniqueName: \"kubernetes.io/projected/04b215da-5dde-4725-b707-18b8aa4737ad-kube-api-access-krrm5\") pod \"payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g\" (UID: \"04b215da-5dde-4725-b707-18b8aa4737ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:26.944649 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.944617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b215da-5dde-4725-b707-18b8aa4737ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g\" (UID: \"04b215da-5dde-4725-b707-18b8aa4737ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:26.944836 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.944654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvfc\" (UniqueName: \"kubernetes.io/projected/41dbbc29-ac14-4cf6-86ef-03c317976256-kube-api-access-bsvfc\") pod \"payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc\" (UID: \"41dbbc29-ac14-4cf6-86ef-03c317976256\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:26.944910 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.944829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrm5\" (UniqueName: \"kubernetes.io/projected/04b215da-5dde-4725-b707-18b8aa4737ad-kube-api-access-krrm5\") pod \"payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g\" (UID: \"04b215da-5dde-4725-b707-18b8aa4737ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:26.944977 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.944954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dbbc29-ac14-4cf6-86ef-03c317976256-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc\" (UID: \"41dbbc29-ac14-4cf6-86ef-03c317976256\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:26.945433 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.945416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b215da-5dde-4725-b707-18b8aa4737ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g\" (UID: \"04b215da-5dde-4725-b707-18b8aa4737ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:26.945628 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.945609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dbbc29-ac14-4cf6-86ef-03c317976256-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc\" (UID: \"41dbbc29-ac14-4cf6-86ef-03c317976256\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:26.952202 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.952175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvfc\" (UniqueName: \"kubernetes.io/projected/41dbbc29-ac14-4cf6-86ef-03c317976256-kube-api-access-bsvfc\") pod \"payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc\" (UID: \"41dbbc29-ac14-4cf6-86ef-03c317976256\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:26.952309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:26.952282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrm5\" (UniqueName: \"kubernetes.io/projected/04b215da-5dde-4725-b707-18b8aa4737ad-kube-api-access-krrm5\") pod \"payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g\" (UID: \"04b215da-5dde-4725-b707-18b8aa4737ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:27.076342 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:27.076308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:27.113251 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:27.113219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:27.348508 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:27.348407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"]
Sep 08 07:24:27.351145 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:24:27.351113    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b215da_5dde_4725_b707_18b8aa4737ad.slice/crio-8f6eb800e2b939dc1470bf8b17de61e111bd33dedb1591841387cbb6b3ffed4c WatchSource:0}: Error finding container 8f6eb800e2b939dc1470bf8b17de61e111bd33dedb1591841387cbb6b3ffed4c: Status 404 returned error can't find the container with id 8f6eb800e2b939dc1470bf8b17de61e111bd33dedb1591841387cbb6b3ffed4c
Sep 08 07:24:27.395384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:27.395362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"]
Sep 08 07:24:27.398106 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:24:27.398077    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41dbbc29_ac14_4cf6_86ef_03c317976256.slice/crio-6ed93338e360478ddc459293cb8b9dfc1c7ad38a480774a22ca8e6c8f62cb575 WatchSource:0}: Error finding container 6ed93338e360478ddc459293cb8b9dfc1c7ad38a480774a22ca8e6c8f62cb575: Status 404 returned error can't find the container with id 6ed93338e360478ddc459293cb8b9dfc1c7ad38a480774a22ca8e6c8f62cb575
Sep 08 07:24:27.642071 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:27.641977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:27.741681 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:27.741639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:28.332681 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.332646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g" event={"ID":"04b215da-5dde-4725-b707-18b8aa4737ad","Type":"ContainerStarted","Data":"0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c"}
Sep 08 07:24:28.332877 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.332699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g" event={"ID":"04b215da-5dde-4725-b707-18b8aa4737ad","Type":"ContainerStarted","Data":"8f6eb800e2b939dc1470bf8b17de61e111bd33dedb1591841387cbb6b3ffed4c"}
Sep 08 07:24:28.332877 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.332731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:28.334292 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.334269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc" event={"ID":"41dbbc29-ac14-4cf6-86ef-03c317976256","Type":"ContainerStarted","Data":"fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683"}
Sep 08 07:24:28.334411 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.334306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc" event={"ID":"41dbbc29-ac14-4cf6-86ef-03c317976256","Type":"ContainerStarted","Data":"6ed93338e360478ddc459293cb8b9dfc1c7ad38a480774a22ca8e6c8f62cb575"}
Sep 08 07:24:28.334411 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.334326    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:28.348971 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.348926    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g" podStartSLOduration=2.3489095190000002 podStartE2EDuration="2.348909519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:24:28.34731658 +0000 UTC m=+1184.030776748" watchObservedRunningTime="2026-09-08 07:24:28.348909519 +0000 UTC m=+1184.032369841"
Sep 08 07:24:28.362403 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.362358    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc" podStartSLOduration=2.362340194 podStartE2EDuration="2.362340194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:24:28.361246323 +0000 UTC m=+1184.044706477" watchObservedRunningTime="2026-09-08 07:24:28.362340194 +0000 UTC m=+1184.045800346"
Sep 08 07:24:28.801799 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.801715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:28.901393 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.901356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:28.954263 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:28.954223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:29.354617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:29.354585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:29.436830 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:29.436795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:39.342382 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:39.342348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:24:39.342862 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:39.342417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:24:47.076689 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:47.076645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:49.755634 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:49.755599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:49.920138 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:49.920106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:50.521597 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:50.521564    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:50.526000 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:50.525979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:50.528554 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:50.528519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lxxzx\""
Sep 08 07:24:50.530531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:50.530497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:50.665453 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:50.665409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64n8\" (UniqueName: \"kubernetes.io/projected/8586b852-c5ba-42d6-9b03-6b1a49720c19-kube-api-access-n64n8\") pod \"test-curl-7885-140171059721536\" (UID: \"8586b852-c5ba-42d6-9b03-6b1a49720c19\") " pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:50.766838 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:50.766798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64n8\" (UniqueName: \"kubernetes.io/projected/8586b852-c5ba-42d6-9b03-6b1a49720c19-kube-api-access-n64n8\") pod \"test-curl-7885-140171059721536\" (UID: \"8586b852-c5ba-42d6-9b03-6b1a49720c19\") " pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:50.774300 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:50.774226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64n8\" (UniqueName: \"kubernetes.io/projected/8586b852-c5ba-42d6-9b03-6b1a49720c19-kube-api-access-n64n8\") pod \"test-curl-7885-140171059721536\" (UID: \"8586b852-c5ba-42d6-9b03-6b1a49720c19\") " pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:50.837004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:50.836968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:51.105252 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:51.105220    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:51.108489 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:24:51.108447    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8586b852_c5ba_42d6_9b03_6b1a49720c19.slice/crio-5679aab868fac949ad285c288b5fcdabf338ad9a965fb2190b0f86eb06b151c6 WatchSource:0}: Error finding container 5679aab868fac949ad285c288b5fcdabf338ad9a965fb2190b0f86eb06b151c6: Status 404 returned error can't find the container with id 5679aab868fac949ad285c288b5fcdabf338ad9a965fb2190b0f86eb06b151c6
Sep 08 07:24:51.110309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:51.110289    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 07:24:51.448390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:51.448300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-140171059721536" event={"ID":"8586b852-c5ba-42d6-9b03-6b1a49720c19","Type":"ContainerStarted","Data":"5679aab868fac949ad285c288b5fcdabf338ad9a965fb2190b0f86eb06b151c6"}
Sep 08 07:24:52.532818 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:52.532781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:24:53.459005 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:53.458970    2580 generic.go:356] "Generic (PLEG): container finished" podID="8586b852-c5ba-42d6-9b03-6b1a49720c19" containerID="1355d8a96570d8ebe6bdff0ae8a1f3434525e492b31820ec1d7d2d20a7990c60" exitCode=0
Sep 08 07:24:53.459005 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:53.459008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-140171059721536" event={"ID":"8586b852-c5ba-42d6-9b03-6b1a49720c19","Type":"ContainerDied","Data":"1355d8a96570d8ebe6bdff0ae8a1f3434525e492b31820ec1d7d2d20a7990c60"}
Sep 08 07:24:54.593620 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.593595    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:54.612578 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.612554    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7885-140171059721536_8586b852-c5ba-42d6-9b03-6b1a49720c19/test-curl-7885-140171059721536/0.log"
Sep 08 07:24:54.632867 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.632835    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:54.636584 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.636547    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:54.703914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.703877    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n64n8\" (UniqueName: \"kubernetes.io/projected/8586b852-c5ba-42d6-9b03-6b1a49720c19-kube-api-access-n64n8\") pod \"8586b852-c5ba-42d6-9b03-6b1a49720c19\" (UID: \"8586b852-c5ba-42d6-9b03-6b1a49720c19\") "
Sep 08 07:24:54.706053 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.706021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8586b852-c5ba-42d6-9b03-6b1a49720c19-kube-api-access-n64n8" (OuterVolumeSpecName: "kube-api-access-n64n8") pod "8586b852-c5ba-42d6-9b03-6b1a49720c19" (UID: "8586b852-c5ba-42d6-9b03-6b1a49720c19"). InnerVolumeSpecName "kube-api-access-n64n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:24:54.750669 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.750582    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:54.750995 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.750982    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8586b852-c5ba-42d6-9b03-6b1a49720c19" containerName="test-curl-7885-140171059721536"
Sep 08 07:24:54.751041 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.750996    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8586b852-c5ba-42d6-9b03-6b1a49720c19" containerName="test-curl-7885-140171059721536"
Sep 08 07:24:54.751076 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.751054    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8586b852-c5ba-42d6-9b03-6b1a49720c19" containerName="test-curl-7885-140171059721536"
Sep 08 07:24:54.754230 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.754213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:54.759655 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.759624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:54.805254 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.805220    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64n8\" (UniqueName: \"kubernetes.io/projected/8586b852-c5ba-42d6-9b03-6b1a49720c19-kube-api-access-n64n8\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:24:54.905857 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.905819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppsrl\" (UniqueName: \"kubernetes.io/projected/5f6697fc-0428-464f-af38-ad4b31ac2451-kube-api-access-ppsrl\") pod \"test-curl-7885-140171059721536\" (UID: \"5f6697fc-0428-464f-af38-ad4b31ac2451\") " pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:54.940952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:54.940914    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8586b852-c5ba-42d6-9b03-6b1a49720c19" path="/var/lib/kubelet/pods/8586b852-c5ba-42d6-9b03-6b1a49720c19/volumes"
Sep 08 07:24:55.007212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:55.007111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppsrl\" (UniqueName: \"kubernetes.io/projected/5f6697fc-0428-464f-af38-ad4b31ac2451-kube-api-access-ppsrl\") pod \"test-curl-7885-140171059721536\" (UID: \"5f6697fc-0428-464f-af38-ad4b31ac2451\") " pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:55.014898 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:55.014865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppsrl\" (UniqueName: \"kubernetes.io/projected/5f6697fc-0428-464f-af38-ad4b31ac2451-kube-api-access-ppsrl\") pod \"test-curl-7885-140171059721536\" (UID: \"5f6697fc-0428-464f-af38-ad4b31ac2451\") " pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:55.064886 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:55.064860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:55.334162 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:55.334137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:55.336361 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:24:55.336333    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f6697fc_0428_464f_af38_ad4b31ac2451.slice/crio-41721aba0960a5e0f19fe9ee6398a2cd6ceb056369a7749c80327863eb9fe57c WatchSource:0}: Error finding container 41721aba0960a5e0f19fe9ee6398a2cd6ceb056369a7749c80327863eb9fe57c: Status 404 returned error can't find the container with id 41721aba0960a5e0f19fe9ee6398a2cd6ceb056369a7749c80327863eb9fe57c
Sep 08 07:24:55.470353 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:55.470318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-140171059721536" event={"ID":"5f6697fc-0428-464f-af38-ad4b31ac2451","Type":"ContainerStarted","Data":"41721aba0960a5e0f19fe9ee6398a2cd6ceb056369a7749c80327863eb9fe57c"}
Sep 08 07:24:55.471646 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:55.471626    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:55.471774 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:55.471628    2580 scope.go:117] "RemoveContainer" containerID="1355d8a96570d8ebe6bdff0ae8a1f3434525e492b31820ec1d7d2d20a7990c60"
Sep 08 07:24:57.484174 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:57.484132    2580 generic.go:356] "Generic (PLEG): container finished" podID="5f6697fc-0428-464f-af38-ad4b31ac2451" containerID="70af080a2cc257d810be464112f0148509dee93dd8f91e8f20e43bd4482d78ba" exitCode=0
Sep 08 07:24:57.484598 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:57.484187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-140171059721536" event={"ID":"5f6697fc-0428-464f-af38-ad4b31ac2451","Type":"ContainerDied","Data":"70af080a2cc257d810be464112f0148509dee93dd8f91e8f20e43bd4482d78ba"}
Sep 08 07:24:58.611559 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.611535    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:58.630782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.630760    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7885-140171059721536_5f6697fc-0428-464f-af38-ad4b31ac2451/test-curl-7885-140171059721536/0.log"
Sep 08 07:24:58.651246 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.651215    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:58.657263 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.657238    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7885-140171059721536"]
Sep 08 07:24:58.744857 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.744763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppsrl\" (UniqueName: \"kubernetes.io/projected/5f6697fc-0428-464f-af38-ad4b31ac2451-kube-api-access-ppsrl\") pod \"5f6697fc-0428-464f-af38-ad4b31ac2451\" (UID: \"5f6697fc-0428-464f-af38-ad4b31ac2451\") "
Sep 08 07:24:58.746874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.746848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6697fc-0428-464f-af38-ad4b31ac2451-kube-api-access-ppsrl" (OuterVolumeSpecName: "kube-api-access-ppsrl") pod "5f6697fc-0428-464f-af38-ad4b31ac2451" (UID: "5f6697fc-0428-464f-af38-ad4b31ac2451"). InnerVolumeSpecName "kube-api-access-ppsrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:24:58.769001 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.768966    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:24:58.769340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.769327    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f6697fc-0428-464f-af38-ad4b31ac2451" containerName="test-curl-7885-140171059721536"
Sep 08 07:24:58.769385 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.769342    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f6697fc-0428-464f-af38-ad4b31ac2451" containerName="test-curl-7885-140171059721536"
Sep 08 07:24:58.769424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.769408    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f6697fc-0428-464f-af38-ad4b31ac2451" containerName="test-curl-7885-140171059721536"
Sep 08 07:24:58.772241 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.772224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:24:58.777773 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.777748    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:24:58.845575 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.845535    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppsrl\" (UniqueName: \"kubernetes.io/projected/5f6697fc-0428-464f-af38-ad4b31ac2451-kube-api-access-ppsrl\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:24:58.940328 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.940299    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f6697fc-0428-464f-af38-ad4b31ac2451" path="/var/lib/kubelet/pods/5f6697fc-0428-464f-af38-ad4b31ac2451/volumes"
Sep 08 07:24:58.945990 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:58.945967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68v28\" (UniqueName: \"kubernetes.io/projected/74fc3302-d1f0-4376-97bd-e551f116769b-kube-api-access-68v28\") pod \"test-curl-7885-140171059722112\" (UID: \"74fc3302-d1f0-4376-97bd-e551f116769b\") " pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:24:59.047220 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:59.047118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68v28\" (UniqueName: \"kubernetes.io/projected/74fc3302-d1f0-4376-97bd-e551f116769b-kube-api-access-68v28\") pod \"test-curl-7885-140171059722112\" (UID: \"74fc3302-d1f0-4376-97bd-e551f116769b\") " pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:24:59.055382 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:59.055358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68v28\" (UniqueName: \"kubernetes.io/projected/74fc3302-d1f0-4376-97bd-e551f116769b-kube-api-access-68v28\") pod \"test-curl-7885-140171059722112\" (UID: \"74fc3302-d1f0-4376-97bd-e551f116769b\") " pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:24:59.083066 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:59.083032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:24:59.350543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:59.350516    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:24:59.353367 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:24:59.353336    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74fc3302_d1f0_4376_97bd_e551f116769b.slice/crio-a1f9fcea96d87e189998efc0e210d732bc5c44b54c9c3bd07433a46aa6a60bfa WatchSource:0}: Error finding container a1f9fcea96d87e189998efc0e210d732bc5c44b54c9c3bd07433a46aa6a60bfa: Status 404 returned error can't find the container with id a1f9fcea96d87e189998efc0e210d732bc5c44b54c9c3bd07433a46aa6a60bfa
Sep 08 07:24:59.494822 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:59.494790    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059721536"
Sep 08 07:24:59.495015 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:59.494837    2580 scope.go:117] "RemoveContainer" containerID="70af080a2cc257d810be464112f0148509dee93dd8f91e8f20e43bd4482d78ba"
Sep 08 07:24:59.496226 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:24:59.496199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-140171059722112" event={"ID":"74fc3302-d1f0-4376-97bd-e551f116769b","Type":"ContainerStarted","Data":"a1f9fcea96d87e189998efc0e210d732bc5c44b54c9c3bd07433a46aa6a60bfa"}
Sep 08 07:25:00.847641 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:00.847594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:01.510693 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:01.510660    2580 generic.go:356] "Generic (PLEG): container finished" podID="74fc3302-d1f0-4376-97bd-e551f116769b" containerID="8dcc8d1168c46abed9a8085faa8491e5c2b2b1ae5dfc355075fc58669f895ef2" exitCode=0
Sep 08 07:25:01.512850 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:01.510748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-140171059722112" event={"ID":"74fc3302-d1f0-4376-97bd-e551f116769b","Type":"ContainerDied","Data":"8dcc8d1168c46abed9a8085faa8491e5c2b2b1ae5dfc355075fc58669f895ef2"}
Sep 08 07:25:02.642556 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.642521    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:02.661729 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.661704    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7885-140171059722112_74fc3302-d1f0-4376-97bd-e551f116769b/test-curl-7885-140171059722112/0.log"
Sep 08 07:25:02.684158 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.684120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:25:02.687809 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.687785    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:25:02.783897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.783817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68v28\" (UniqueName: \"kubernetes.io/projected/74fc3302-d1f0-4376-97bd-e551f116769b-kube-api-access-68v28\") pod \"74fc3302-d1f0-4376-97bd-e551f116769b\" (UID: \"74fc3302-d1f0-4376-97bd-e551f116769b\") "
Sep 08 07:25:02.785942 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.785916    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fc3302-d1f0-4376-97bd-e551f116769b-kube-api-access-68v28" (OuterVolumeSpecName: "kube-api-access-68v28") pod "74fc3302-d1f0-4376-97bd-e551f116769b" (UID: "74fc3302-d1f0-4376-97bd-e551f116769b"). InnerVolumeSpecName "kube-api-access-68v28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:25:02.797420 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.797394    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:25:02.797833 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.797817    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fc3302-d1f0-4376-97bd-e551f116769b" containerName="test-curl-7885-140171059722112"
Sep 08 07:25:02.797878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.797835    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fc3302-d1f0-4376-97bd-e551f116769b" containerName="test-curl-7885-140171059722112"
Sep 08 07:25:02.797914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.797898    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fc3302-d1f0-4376-97bd-e551f116769b" containerName="test-curl-7885-140171059722112"
Sep 08 07:25:02.800677 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.800662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:02.807051 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.807021    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:25:02.885138 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.885097    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68v28\" (UniqueName: \"kubernetes.io/projected/74fc3302-d1f0-4376-97bd-e551f116769b-kube-api-access-68v28\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:25:02.940750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.940707    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74fc3302-d1f0-4376-97bd-e551f116769b" path="/var/lib/kubelet/pods/74fc3302-d1f0-4376-97bd-e551f116769b/volumes"
Sep 08 07:25:02.985984 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:02.985941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6ml\" (UniqueName: \"kubernetes.io/projected/6dd8c23d-2bc6-4c4e-b542-1046cc6958af-kube-api-access-sn6ml\") pod \"test-curl-7885-140171059722112\" (UID: \"6dd8c23d-2bc6-4c4e-b542-1046cc6958af\") " pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:03.087192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:03.087153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6ml\" (UniqueName: \"kubernetes.io/projected/6dd8c23d-2bc6-4c4e-b542-1046cc6958af-kube-api-access-sn6ml\") pod \"test-curl-7885-140171059722112\" (UID: \"6dd8c23d-2bc6-4c4e-b542-1046cc6958af\") " pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:03.094712 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:03.094676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6ml\" (UniqueName: \"kubernetes.io/projected/6dd8c23d-2bc6-4c4e-b542-1046cc6958af-kube-api-access-sn6ml\") pod \"test-curl-7885-140171059722112\" (UID: \"6dd8c23d-2bc6-4c4e-b542-1046cc6958af\") " pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:03.111265 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:03.111228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:03.382749 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:03.382612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:25:03.385112 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:25:03.385080    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dd8c23d_2bc6_4c4e_b542_1046cc6958af.slice/crio-fa9846761318ecad902cf8b3724bbd40b3338aad9faa9f505d45674ff0913caa WatchSource:0}: Error finding container fa9846761318ecad902cf8b3724bbd40b3338aad9faa9f505d45674ff0913caa: Status 404 returned error can't find the container with id fa9846761318ecad902cf8b3724bbd40b3338aad9faa9f505d45674ff0913caa
Sep 08 07:25:03.521958 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:03.521912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-140171059722112" event={"ID":"6dd8c23d-2bc6-4c4e-b542-1046cc6958af","Type":"ContainerStarted","Data":"fa9846761318ecad902cf8b3724bbd40b3338aad9faa9f505d45674ff0913caa"}
Sep 08 07:25:03.523163 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:03.523142    2580 scope.go:117] "RemoveContainer" containerID="8dcc8d1168c46abed9a8085faa8491e5c2b2b1ae5dfc355075fc58669f895ef2"
Sep 08 07:25:03.523163 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:03.523156    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:04.770107 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:04.770038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:04.821767 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:04.821727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:05.536179 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:05.536132    2580 generic.go:356] "Generic (PLEG): container finished" podID="6dd8c23d-2bc6-4c4e-b542-1046cc6958af" containerID="cacbc1891dce4fdb9f6d9f3f9acd7db49bd487490bb666e9829b74e62804c82e" exitCode=0
Sep 08 07:25:05.536179 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:05.536181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-140171059722112" event={"ID":"6dd8c23d-2bc6-4c4e-b542-1046cc6958af","Type":"ContainerDied","Data":"cacbc1891dce4fdb9f6d9f3f9acd7db49bd487490bb666e9829b74e62804c82e"}
Sep 08 07:25:06.676192 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:06.676168    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:06.694632 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:06.694607    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7885-140171059722112_6dd8c23d-2bc6-4c4e-b542-1046cc6958af/test-curl-7885-140171059722112/0.log"
Sep 08 07:25:06.714989 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:06.714959    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:25:06.717990 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:06.717962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn6ml\" (UniqueName: \"kubernetes.io/projected/6dd8c23d-2bc6-4c4e-b542-1046cc6958af-kube-api-access-sn6ml\") pod \"6dd8c23d-2bc6-4c4e-b542-1046cc6958af\" (UID: \"6dd8c23d-2bc6-4c4e-b542-1046cc6958af\") "
Sep 08 07:25:06.720118 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:06.720089    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7885-140171059722112"]
Sep 08 07:25:06.720368 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:06.720342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd8c23d-2bc6-4c4e-b542-1046cc6958af-kube-api-access-sn6ml" (OuterVolumeSpecName: "kube-api-access-sn6ml") pod "6dd8c23d-2bc6-4c4e-b542-1046cc6958af" (UID: "6dd8c23d-2bc6-4c4e-b542-1046cc6958af"). InnerVolumeSpecName "kube-api-access-sn6ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:25:06.819252 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:06.819222    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn6ml\" (UniqueName: \"kubernetes.io/projected/6dd8c23d-2bc6-4c4e-b542-1046cc6958af-kube-api-access-sn6ml\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:25:06.940991 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:06.940956    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dd8c23d-2bc6-4c4e-b542-1046cc6958af" path="/var/lib/kubelet/pods/6dd8c23d-2bc6-4c4e-b542-1046cc6958af/volumes"
Sep 08 07:25:07.289890 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:07.289777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:07.549868 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:07.549787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-140171059722112"
Sep 08 07:25:07.549868 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:07.549847    2580 scope.go:117] "RemoveContainer" containerID="cacbc1891dce4fdb9f6d9f3f9acd7db49bd487490bb666e9829b74e62804c82e"
Sep 08 07:25:07.577394 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:07.577363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:07.864866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:07.864836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:10.296899 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:10.296864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:10.390134 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:10.390100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:11.135171 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.135140    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"]
Sep 08 07:25:11.135535 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.135520    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd8c23d-2bc6-4c4e-b542-1046cc6958af" containerName="test-curl-7885-140171059722112"
Sep 08 07:25:11.135594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.135536    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd8c23d-2bc6-4c4e-b542-1046cc6958af" containerName="test-curl-7885-140171059722112"
Sep 08 07:25:11.135631 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.135599    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd8c23d-2bc6-4c4e-b542-1046cc6958af" containerName="test-curl-7885-140171059722112"
Sep 08 07:25:11.140137 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.140113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.142909 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.142866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"auth-test-model-fb59c2-kserve-self-signed-certs\""
Sep 08 07:25:11.143050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.142912    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"openshift-service-ca.crt\""
Sep 08 07:25:11.143050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.142878    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"kube-root-ca.crt\""
Sep 08 07:25:11.143050 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.142894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"default-dockercfg-ltj4x\""
Sep 08 07:25:11.150713 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.150681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"]
Sep 08 07:25:11.153267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.153246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-tmp-dir\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.153367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.153274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-home\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.153367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.153310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73db49b-6999-4938-964e-a02dcfd51652-tls-certs\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.153440 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.153393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-model-cache\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.153491 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.153435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-dshm\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.153537 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.153495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzkwv\" (UniqueName: \"kubernetes.io/projected/b73db49b-6999-4938-964e-a02dcfd51652-kube-api-access-jzkwv\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.254411 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73db49b-6999-4938-964e-a02dcfd51652-tls-certs\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.254618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-model-cache\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.254618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-dshm\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.254618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkwv\" (UniqueName: \"kubernetes.io/projected/b73db49b-6999-4938-964e-a02dcfd51652-kube-api-access-jzkwv\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.254618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-tmp-dir\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.254618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-home\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.254907 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-model-cache\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.254907 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-tmp-dir\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.255067 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.254950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-home\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.256749 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.256729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-dshm\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.257093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.257069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73db49b-6999-4938-964e-a02dcfd51652-tls-certs\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.262978 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.262956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzkwv\" (UniqueName: \"kubernetes.io/projected/b73db49b-6999-4938-964e-a02dcfd51652-kube-api-access-jzkwv\") pod \"auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.439901 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.439813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:11.449082 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.449050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:11.541582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.541550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:11.727777 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.727701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"]
Sep 08 07:25:11.731520 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:25:11.731489    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb73db49b_6999_4938_964e_a02dcfd51652.slice/crio-af565e0c4fa01d06441bb92c99fde1f09978900b3ea49838ce557dfd5fed04c3 WatchSource:0}: Error finding container af565e0c4fa01d06441bb92c99fde1f09978900b3ea49838ce557dfd5fed04c3: Status 404 returned error can't find the container with id af565e0c4fa01d06441bb92c99fde1f09978900b3ea49838ce557dfd5fed04c3
Sep 08 07:25:11.733262 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.733238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:11.972599 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:11.972545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:12.037565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.037450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:12.060228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.060196    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"]
Sep 08 07:25:12.063730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.063708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:12.065568 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.065550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-15977b5c\""
Sep 08 07:25:12.070784 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.070755    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"]
Sep 08 07:25:12.097780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.097748    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"]
Sep 08 07:25:12.102979 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.102958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:12.104974 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.104956    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-15977b5c-dockercfg-9qzl9\""
Sep 08 07:25:12.110275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.110251    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"]
Sep 08 07:25:12.161783 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.161742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqmlr\" (UniqueName: \"kubernetes.io/projected/36481b33-db3b-499c-b05f-dd307084b68c-kube-api-access-pqmlr\") pod \"payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql\" (UID: \"36481b33-db3b-499c-b05f-dd307084b68c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:12.162028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.161867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36481b33-db3b-499c-b05f-dd307084b68c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql\" (UID: \"36481b33-db3b-499c-b05f-dd307084b68c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:12.262690 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.262650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36481b33-db3b-499c-b05f-dd307084b68c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql\" (UID: \"36481b33-db3b-499c-b05f-dd307084b68c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:12.262873 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.262737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10262d75-c005-470a-b7c3-601064c3d8b8-plugins-config-volume\") pod \"payload-processing-e2e-mt-15977b5c-795957588f-5pjg9\" (UID: \"10262d75-c005-470a-b7c3-601064c3d8b8\") " pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:12.262873 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.262776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qb4h\" (UniqueName: \"kubernetes.io/projected/10262d75-c005-470a-b7c3-601064c3d8b8-kube-api-access-5qb4h\") pod \"payload-processing-e2e-mt-15977b5c-795957588f-5pjg9\" (UID: \"10262d75-c005-470a-b7c3-601064c3d8b8\") " pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:12.262873 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.262824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqmlr\" (UniqueName: \"kubernetes.io/projected/36481b33-db3b-499c-b05f-dd307084b68c-kube-api-access-pqmlr\") pod \"payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql\" (UID: \"36481b33-db3b-499c-b05f-dd307084b68c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:12.263634 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.263595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36481b33-db3b-499c-b05f-dd307084b68c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql\" (UID: \"36481b33-db3b-499c-b05f-dd307084b68c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:12.270735 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.270713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqmlr\" (UniqueName: \"kubernetes.io/projected/36481b33-db3b-499c-b05f-dd307084b68c-kube-api-access-pqmlr\") pod \"payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql\" (UID: \"36481b33-db3b-499c-b05f-dd307084b68c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:12.364492 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.364422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10262d75-c005-470a-b7c3-601064c3d8b8-plugins-config-volume\") pod \"payload-processing-e2e-mt-15977b5c-795957588f-5pjg9\" (UID: \"10262d75-c005-470a-b7c3-601064c3d8b8\") " pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:12.364671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.364535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qb4h\" (UniqueName: \"kubernetes.io/projected/10262d75-c005-470a-b7c3-601064c3d8b8-kube-api-access-5qb4h\") pod \"payload-processing-e2e-mt-15977b5c-795957588f-5pjg9\" (UID: \"10262d75-c005-470a-b7c3-601064c3d8b8\") " pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:12.365818 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.365784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10262d75-c005-470a-b7c3-601064c3d8b8-plugins-config-volume\") pod \"payload-processing-e2e-mt-15977b5c-795957588f-5pjg9\" (UID: \"10262d75-c005-470a-b7c3-601064c3d8b8\") " pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:12.373032 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.373005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qb4h\" (UniqueName: \"kubernetes.io/projected/10262d75-c005-470a-b7c3-601064c3d8b8-kube-api-access-5qb4h\") pod \"payload-processing-e2e-mt-15977b5c-795957588f-5pjg9\" (UID: \"10262d75-c005-470a-b7c3-601064c3d8b8\") " pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:12.374632 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.374612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:12.414999 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.414962    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:12.576081 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.576048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl" event={"ID":"b73db49b-6999-4938-964e-a02dcfd51652","Type":"ContainerStarted","Data":"1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966"}
Sep 08 07:25:12.576482 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.576092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl" event={"ID":"b73db49b-6999-4938-964e-a02dcfd51652","Type":"ContainerStarted","Data":"af565e0c4fa01d06441bb92c99fde1f09978900b3ea49838ce557dfd5fed04c3"}
Sep 08 07:25:12.592158 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.592102    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl" podStartSLOduration=1.592086015 podStartE2EDuration="1.592086015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:25:12.590357594 +0000 UTC m=+1228.273817747" watchObservedRunningTime="2026-09-08 07:25:12.592086015 +0000 UTC m=+1228.275546167"
Sep 08 07:25:12.645414 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.645389    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"]
Sep 08 07:25:12.647423 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:25:12.647394    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36481b33_db3b_499c_b05f_dd307084b68c.slice/crio-e49be8a3e20b3eb73714122970d58e22ebd12e4edea11bbf1bd5573a36390f26 WatchSource:0}: Error finding container e49be8a3e20b3eb73714122970d58e22ebd12e4edea11bbf1bd5573a36390f26: Status 404 returned error can't find the container with id e49be8a3e20b3eb73714122970d58e22ebd12e4edea11bbf1bd5573a36390f26
Sep 08 07:25:12.696049 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:12.696027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"]
Sep 08 07:25:12.698829 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:25:12.698800    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10262d75_c005_470a_b7c3_601064c3d8b8.slice/crio-a411f68948a088fed15e9ee22c88d395bafbf8f4f66f678d0a4dee64764da604 WatchSource:0}: Error finding container a411f68948a088fed15e9ee22c88d395bafbf8f4f66f678d0a4dee64764da604: Status 404 returned error can't find the container with id a411f68948a088fed15e9ee22c88d395bafbf8f4f66f678d0a4dee64764da604
Sep 08 07:25:13.582900 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:13.582859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9" event={"ID":"10262d75-c005-470a-b7c3-601064c3d8b8","Type":"ContainerStarted","Data":"cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa"}
Sep 08 07:25:13.583389 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:13.583029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:13.583389 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:13.583049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9" event={"ID":"10262d75-c005-470a-b7c3-601064c3d8b8","Type":"ContainerStarted","Data":"a411f68948a088fed15e9ee22c88d395bafbf8f4f66f678d0a4dee64764da604"}
Sep 08 07:25:13.584499 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:13.584474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql" event={"ID":"36481b33-db3b-499c-b05f-dd307084b68c","Type":"ContainerStarted","Data":"9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7"}
Sep 08 07:25:13.584574 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:13.584521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql" event={"ID":"36481b33-db3b-499c-b05f-dd307084b68c","Type":"ContainerStarted","Data":"e49be8a3e20b3eb73714122970d58e22ebd12e4edea11bbf1bd5573a36390f26"}
Sep 08 07:25:13.584620 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:13.584598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:13.600367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:13.600313    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9" podStartSLOduration=1.600295644 podStartE2EDuration="1.600295644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:25:13.598480371 +0000 UTC m=+1229.281940522" watchObservedRunningTime="2026-09-08 07:25:13.600295644 +0000 UTC m=+1229.283755797"
Sep 08 07:25:13.614612 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:13.614559    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql" podStartSLOduration=1.6145422699999998 podStartE2EDuration="1.61454227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:25:13.612246321 +0000 UTC m=+1229.295706485" watchObservedRunningTime="2026-09-08 07:25:13.61454227 +0000 UTC m=+1229.298002422"
Sep 08 07:25:21.449736 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:21.449695    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:21.449736 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:21.449741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:21.462156 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:21.462125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:21.637997 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:21.637970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:25:24.591664 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:24.591631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:25:24.592115 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:24.591817    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:25:26.953530 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:26.953494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:32.356659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:32.356623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:35.231609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:35.231573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:44.950735 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:44.950668    2580 scope.go:117] "RemoveContainer" containerID="5fd4983c812aeef4aabece272956d4d323598f4d3399042bdb15d86f98bcf872"
Sep 08 07:25:45.236569 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.236487    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"]
Sep 08 07:25:45.241233 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.241210    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.243231 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.243208    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"openshift-service-ca.crt\""
Sep 08 07:25:45.244060 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.244041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"default-dockercfg-r9gnf\""
Sep 08 07:25:45.244197 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.244062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"auth-test-model-217bc2-kserve-self-signed-certs\""
Sep 08 07:25:45.244294 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.244066    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"kube-root-ca.crt\""
Sep 08 07:25:45.248954 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.248929    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"]
Sep 08 07:25:45.273078 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.273042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:25:45.376328 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.376289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdlr\" (UniqueName: \"kubernetes.io/projected/0625353e-87c5-49c7-87ab-e258844c6b32-kube-api-access-ngdlr\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.376539 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.376341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-home\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.376539 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.376507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0625353e-87c5-49c7-87ab-e258844c6b32-tls-certs\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.376635 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.376545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-model-cache\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.376635 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.376563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-dshm\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.376635 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.376622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-tmp-dir\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.477419 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.477377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-tmp-dir\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.477419 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.477427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdlr\" (UniqueName: \"kubernetes.io/projected/0625353e-87c5-49c7-87ab-e258844c6b32-kube-api-access-ngdlr\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.477713 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.477484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-home\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.477713 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.477532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0625353e-87c5-49c7-87ab-e258844c6b32-tls-certs\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.477713 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.477555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-model-cache\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.477713 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.477572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-dshm\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.484136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.478140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-home\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.484136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.478214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-tmp-dir\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.484136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.478360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-model-cache\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.484136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.480714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-dshm\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.484136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.480985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0625353e-87c5-49c7-87ab-e258844c6b32-tls-certs\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.487168 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.487107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdlr\" (UniqueName: \"kubernetes.io/projected/0625353e-87c5-49c7-87ab-e258844c6b32-kube-api-access-ngdlr\") pod \"auth-test-model-217bc2-kserve-6dc7796fb7-crrl5\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.553317 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.553268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:45.825510 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:45.825480    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"]
Sep 08 07:25:45.827888 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:25:45.827860    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0625353e_87c5_49c7_87ab_e258844c6b32.slice/crio-1dddac1d736008e2fb96b6fb0acbad1c9d67af33c55d1f295246f5714225e937 WatchSource:0}: Error finding container 1dddac1d736008e2fb96b6fb0acbad1c9d67af33c55d1f295246f5714225e937: Status 404 returned error can't find the container with id 1dddac1d736008e2fb96b6fb0acbad1c9d67af33c55d1f295246f5714225e937
Sep 08 07:25:46.754790 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:46.754754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5" event={"ID":"0625353e-87c5-49c7-87ab-e258844c6b32","Type":"ContainerStarted","Data":"9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93"}
Sep 08 07:25:46.755196 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:46.754799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5" event={"ID":"0625353e-87c5-49c7-87ab-e258844c6b32","Type":"ContainerStarted","Data":"1dddac1d736008e2fb96b6fb0acbad1c9d67af33c55d1f295246f5714225e937"}
Sep 08 07:25:46.771652 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:46.771597    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5" podStartSLOduration=1.771579688 podStartE2EDuration="1.771579688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:25:46.769430141 +0000 UTC m=+1262.452890295" watchObservedRunningTime="2026-09-08 07:25:46.771579688 +0000 UTC m=+1262.455039872"
Sep 08 07:25:55.553778 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:55.553742    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:55.554372 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:55.553795    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:55.566354 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:55.566321    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:55.818628 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:55.818607    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:25:59.364938 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:25:59.364904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:26:29.819653 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:29.819613    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"]
Sep 08 07:26:29.823672 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:29.823647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"
Sep 08 07:26:29.825571 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:29.825551    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:26:29.825674 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:29.825636    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:26:29.829852 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:29.829828    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"]
Sep 08 07:26:29.994164 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:29.994122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eac734e-e2c1-4f00-865f-fa773e53ed23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj\" (UID: \"0eac734e-e2c1-4f00-865f-fa773e53ed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"
Sep 08 07:26:30.095346 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:30.095267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eac734e-e2c1-4f00-865f-fa773e53ed23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj\" (UID: \"0eac734e-e2c1-4f00-865f-fa773e53ed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"
Sep 08 07:26:30.095903 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:30.095882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eac734e-e2c1-4f00-865f-fa773e53ed23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj\" (UID: \"0eac734e-e2c1-4f00-865f-fa773e53ed23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"
Sep 08 07:26:30.136010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:30.135973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"
Sep 08 07:26:30.406432 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:30.406408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"]
Sep 08 07:26:30.409171 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:26:30.409148    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eac734e_e2c1_4f00_865f_fa773e53ed23.slice/crio-34307edfedbb53ef1bb532f3e5479ee260dd028fbaf87177584009897385fa08 WatchSource:0}: Error finding container 34307edfedbb53ef1bb532f3e5479ee260dd028fbaf87177584009897385fa08: Status 404 returned error can't find the container with id 34307edfedbb53ef1bb532f3e5479ee260dd028fbaf87177584009897385fa08
Sep 08 07:26:30.975510 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:30.975475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj" event={"ID":"0eac734e-e2c1-4f00-865f-fa773e53ed23","Type":"ContainerStarted","Data":"7213653edf8028d010062701d2ec0d608da7677afcc9d1d90268b874c525e645"}
Sep 08 07:26:30.975907 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:30.975526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj" event={"ID":"0eac734e-e2c1-4f00-865f-fa773e53ed23","Type":"ContainerStarted","Data":"34307edfedbb53ef1bb532f3e5479ee260dd028fbaf87177584009897385fa08"}
Sep 08 07:26:30.990408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:30.990355    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj" podStartSLOduration=1.990324172 podStartE2EDuration="1.990324172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:26:30.989318742 +0000 UTC m=+1306.672778896" watchObservedRunningTime="2026-09-08 07:26:30.990324172 +0000 UTC m=+1306.673784325"
Sep 08 07:26:31.983358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:31.983315    2580 generic.go:356] "Generic (PLEG): container finished" podID="0eac734e-e2c1-4f00-865f-fa773e53ed23" containerID="7213653edf8028d010062701d2ec0d608da7677afcc9d1d90268b874c525e645" exitCode=0
Sep 08 07:26:31.983358 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:31.983363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj" event={"ID":"0eac734e-e2c1-4f00-865f-fa773e53ed23","Type":"ContainerDied","Data":"7213653edf8028d010062701d2ec0d608da7677afcc9d1d90268b874c525e645"}
Sep 08 07:26:33.119618 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:33.119591    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"
Sep 08 07:26:33.225074 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:33.225041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eac734e-e2c1-4f00-865f-fa773e53ed23-maas-api-service-ca\") pod \"0eac734e-e2c1-4f00-865f-fa773e53ed23\" (UID: \"0eac734e-e2c1-4f00-865f-fa773e53ed23\") "
Sep 08 07:26:33.225356 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:33.225333    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eac734e-e2c1-4f00-865f-fa773e53ed23-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0eac734e-e2c1-4f00-865f-fa773e53ed23" (UID: "0eac734e-e2c1-4f00-865f-fa773e53ed23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:26:33.326715 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:33.326684    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eac734e-e2c1-4f00-865f-fa773e53ed23-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:26:33.993958 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:33.993923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj" event={"ID":"0eac734e-e2c1-4f00-865f-fa773e53ed23","Type":"ContainerDied","Data":"34307edfedbb53ef1bb532f3e5479ee260dd028fbaf87177584009897385fa08"}
Sep 08 07:26:33.994128 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:33.993976    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34307edfedbb53ef1bb532f3e5479ee260dd028fbaf87177584009897385fa08"
Sep 08 07:26:33.994128 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:33.993931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"
Sep 08 07:26:44.985807 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:44.985772    2580 scope.go:117] "RemoveContainer" containerID="5a82413241b13259ce45e2272db8c63e52e85d9b188eb338604e90a66d2b90bb"
Sep 08 07:26:45.347504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.347451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:26:45.380794 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.380763    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"]
Sep 08 07:26:45.381047 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.381022    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9" podUID="10262d75-c005-470a-b7c3-601064c3d8b8" containerName="payload-processing" containerID="cri-o://cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa" gracePeriod=30
Sep 08 07:26:45.387267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.387240    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"]
Sep 08 07:26:45.387865 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.387648    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql" podUID="36481b33-db3b-499c-b05f-dd307084b68c" containerName="payload-pre-processing" containerID="cri-o://9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7" gracePeriod=30
Sep 08 07:26:45.416293 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.416264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:26:45.483249 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.483218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:26:45.564973 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.564939    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"]
Sep 08 07:26:45.567954 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.567924    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15977b5c-6kpkj"]
Sep 08 07:26:45.568646 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.568617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:26:45.625860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.625780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:26:45.663878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.663854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:26:45.666694 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.666674    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:26:45.741785 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.741753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqmlr\" (UniqueName: \"kubernetes.io/projected/36481b33-db3b-499c-b05f-dd307084b68c-kube-api-access-pqmlr\") pod \"36481b33-db3b-499c-b05f-dd307084b68c\" (UID: \"36481b33-db3b-499c-b05f-dd307084b68c\") "
Sep 08 07:26:45.741785 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.741786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qb4h\" (UniqueName: \"kubernetes.io/projected/10262d75-c005-470a-b7c3-601064c3d8b8-kube-api-access-5qb4h\") pod \"10262d75-c005-470a-b7c3-601064c3d8b8\" (UID: \"10262d75-c005-470a-b7c3-601064c3d8b8\") "
Sep 08 07:26:45.742045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.741867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36481b33-db3b-499c-b05f-dd307084b68c-plugins-config-volume\") pod \"36481b33-db3b-499c-b05f-dd307084b68c\" (UID: \"36481b33-db3b-499c-b05f-dd307084b68c\") "
Sep 08 07:26:45.742045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.741907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10262d75-c005-470a-b7c3-601064c3d8b8-plugins-config-volume\") pod \"10262d75-c005-470a-b7c3-601064c3d8b8\" (UID: \"10262d75-c005-470a-b7c3-601064c3d8b8\") "
Sep 08 07:26:45.742285 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.742254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36481b33-db3b-499c-b05f-dd307084b68c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36481b33-db3b-499c-b05f-dd307084b68c" (UID: "36481b33-db3b-499c-b05f-dd307084b68c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:26:45.742417 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.742322    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10262d75-c005-470a-b7c3-601064c3d8b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10262d75-c005-470a-b7c3-601064c3d8b8" (UID: "10262d75-c005-470a-b7c3-601064c3d8b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:26:45.744012 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.743989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10262d75-c005-470a-b7c3-601064c3d8b8-kube-api-access-5qb4h" (OuterVolumeSpecName: "kube-api-access-5qb4h") pod "10262d75-c005-470a-b7c3-601064c3d8b8" (UID: "10262d75-c005-470a-b7c3-601064c3d8b8"). InnerVolumeSpecName "kube-api-access-5qb4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:26:45.744106 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.744021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36481b33-db3b-499c-b05f-dd307084b68c-kube-api-access-pqmlr" (OuterVolumeSpecName: "kube-api-access-pqmlr") pod "36481b33-db3b-499c-b05f-dd307084b68c" (UID: "36481b33-db3b-499c-b05f-dd307084b68c"). InnerVolumeSpecName "kube-api-access-pqmlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:26:45.843659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.843630    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqmlr\" (UniqueName: \"kubernetes.io/projected/36481b33-db3b-499c-b05f-dd307084b68c-kube-api-access-pqmlr\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:26:45.843659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.843656    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qb4h\" (UniqueName: \"kubernetes.io/projected/10262d75-c005-470a-b7c3-601064c3d8b8-kube-api-access-5qb4h\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:26:45.843659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.843665    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36481b33-db3b-499c-b05f-dd307084b68c-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:26:45.843885 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:45.843674    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10262d75-c005-470a-b7c3-601064c3d8b8-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:26:46.066627 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.066545    2580 generic.go:356] "Generic (PLEG): container finished" podID="36481b33-db3b-499c-b05f-dd307084b68c" containerID="9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7" exitCode=0
Sep 08 07:26:46.066627 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.066589    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"
Sep 08 07:26:46.067106 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.066635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql" event={"ID":"36481b33-db3b-499c-b05f-dd307084b68c","Type":"ContainerDied","Data":"9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7"}
Sep 08 07:26:46.067106 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.066687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql" event={"ID":"36481b33-db3b-499c-b05f-dd307084b68c","Type":"ContainerDied","Data":"e49be8a3e20b3eb73714122970d58e22ebd12e4edea11bbf1bd5573a36390f26"}
Sep 08 07:26:46.067537 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.066715    2580 scope.go:117] "RemoveContainer" containerID="9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7"
Sep 08 07:26:46.068154 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.068130    2580 generic.go:356] "Generic (PLEG): container finished" podID="10262d75-c005-470a-b7c3-601064c3d8b8" containerID="cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa" exitCode=0
Sep 08 07:26:46.068284 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.068194    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"
Sep 08 07:26:46.068284 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.068231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9" event={"ID":"10262d75-c005-470a-b7c3-601064c3d8b8","Type":"ContainerDied","Data":"cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa"}
Sep 08 07:26:46.068403 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.068282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9" event={"ID":"10262d75-c005-470a-b7c3-601064c3d8b8","Type":"ContainerDied","Data":"a411f68948a088fed15e9ee22c88d395bafbf8f4f66f678d0a4dee64764da604"}
Sep 08 07:26:46.077739 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.076999    2580 scope.go:117] "RemoveContainer" containerID="9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7"
Sep 08 07:26:46.079737 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:26:46.079707    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7\": container with ID starting with 9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7 not found: ID does not exist" containerID="9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7"
Sep 08 07:26:46.079855 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.079750    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7"} err="failed to get container status \"9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7\": rpc error: code = NotFound desc = could not find container \"9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7\": container with ID starting with 9ce5945c1e0d251d6983bcf8517cc0008e216d9f2ca57cc7ce2f274359a140b7 not found: ID does not exist"
Sep 08 07:26:46.079855 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.079774    2580 scope.go:117] "RemoveContainer" containerID="cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa"
Sep 08 07:26:46.090984 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.090961    2580 scope.go:117] "RemoveContainer" containerID="cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa"
Sep 08 07:26:46.091281 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:26:46.091257    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa\": container with ID starting with cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa not found: ID does not exist" containerID="cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa"
Sep 08 07:26:46.091427 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.091295    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa"} err="failed to get container status \"cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa\": rpc error: code = NotFound desc = could not find container \"cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa\": container with ID starting with cc27a3f88bfcf4a14f94310600a2b74d4648a62c9a7d37aa9a65d7fe8875adaa not found: ID does not exist"
Sep 08 07:26:46.093839 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.093819    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"]
Sep 08 07:26:46.097243 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.097221    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15977b5c-6b484774fd-jl5ql"]
Sep 08 07:26:46.106521 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.106500    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"]
Sep 08 07:26:46.112152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.112132    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15977b5c-795957588f-5pjg9"]
Sep 08 07:26:46.941081 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.941044    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eac734e-e2c1-4f00-865f-fa773e53ed23" path="/var/lib/kubelet/pods/0eac734e-e2c1-4f00-865f-fa773e53ed23/volumes"
Sep 08 07:26:46.941418 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.941405    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10262d75-c005-470a-b7c3-601064c3d8b8" path="/var/lib/kubelet/pods/10262d75-c005-470a-b7c3-601064c3d8b8/volumes"
Sep 08 07:26:46.941753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:46.941741    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36481b33-db3b-499c-b05f-dd307084b68c" path="/var/lib/kubelet/pods/36481b33-db3b-499c-b05f-dd307084b68c/volumes"
Sep 08 07:26:48.543893 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:48.543863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:26:53.574699 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:26:53.574667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:01.716057 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:01.716003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:29.849134 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:29.849051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:30.888609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:30.888561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:36.640922 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:36.640888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:36.741410 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:36.741363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:36.884970 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:36.884933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:38.112748 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:38.112708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:38.207834 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:38.207797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:38.307726 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:38.307683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:38.853888 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:38.853850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:38.949337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:38.949295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:39.049594 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:39.049559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:40.238878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:40.238844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:46.132208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:46.132163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:46.231415 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:46.231381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:46.798381 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:46.798344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:48.259711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.259673    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7888-88659d"]
Sep 08 07:27:48.260075 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260053    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36481b33-db3b-499c-b05f-dd307084b68c" containerName="payload-pre-processing"
Sep 08 07:27:48.260075 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260065    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="36481b33-db3b-499c-b05f-dd307084b68c" containerName="payload-pre-processing"
Sep 08 07:27:48.260152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260075    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10262d75-c005-470a-b7c3-601064c3d8b8" containerName="payload-processing"
Sep 08 07:27:48.260152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260085    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10262d75-c005-470a-b7c3-601064c3d8b8" containerName="payload-processing"
Sep 08 07:27:48.260152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260101    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eac734e-e2c1-4f00-865f-fa773e53ed23" containerName="revoke-keys"
Sep 08 07:27:48.260152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260107    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eac734e-e2c1-4f00-865f-fa773e53ed23" containerName="revoke-keys"
Sep 08 07:27:48.260285 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260195    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eac734e-e2c1-4f00-865f-fa773e53ed23" containerName="revoke-keys"
Sep 08 07:27:48.260285 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260207    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="36481b33-db3b-499c-b05f-dd307084b68c" containerName="payload-pre-processing"
Sep 08 07:27:48.260285 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.260214    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10262d75-c005-470a-b7c3-601064c3d8b8" containerName="payload-processing"
Sep 08 07:27:48.264762 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.264742    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:48.267046 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.267023    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lxxzx\""
Sep 08 07:27:48.268395 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.268367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7888-88659d"]
Sep 08 07:27:48.439227 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.439191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mkr\" (UniqueName: \"kubernetes.io/projected/84e17bc5-2b17-4ac1-b96c-b328f7219f57-kube-api-access-84mkr\") pod \"mt-curl-7888-88659d\" (UID: \"84e17bc5-2b17-4ac1-b96c-b328f7219f57\") " pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:48.540535 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.540413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mkr\" (UniqueName: \"kubernetes.io/projected/84e17bc5-2b17-4ac1-b96c-b328f7219f57-kube-api-access-84mkr\") pod \"mt-curl-7888-88659d\" (UID: \"84e17bc5-2b17-4ac1-b96c-b328f7219f57\") " pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:48.548775 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.548743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mkr\" (UniqueName: \"kubernetes.io/projected/84e17bc5-2b17-4ac1-b96c-b328f7219f57-kube-api-access-84mkr\") pod \"mt-curl-7888-88659d\" (UID: \"84e17bc5-2b17-4ac1-b96c-b328f7219f57\") " pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:48.574743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.574716    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:48.843298 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:48.843248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7888-88659d"]
Sep 08 07:27:48.845614 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:27:48.845586    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84e17bc5_2b17_4ac1_b96c_b328f7219f57.slice/crio-b5d80901e3f756a149f7b02f182080b17d3fd0938e4b7fa404e75e56e1d4a25c WatchSource:0}: Error finding container b5d80901e3f756a149f7b02f182080b17d3fd0938e4b7fa404e75e56e1d4a25c: Status 404 returned error can't find the container with id b5d80901e3f756a149f7b02f182080b17d3fd0938e4b7fa404e75e56e1d4a25c
Sep 08 07:27:49.387083 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:49.387044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7888-88659d" event={"ID":"84e17bc5-2b17-4ac1-b96c-b328f7219f57","Type":"ContainerStarted","Data":"b5d80901e3f756a149f7b02f182080b17d3fd0938e4b7fa404e75e56e1d4a25c"}
Sep 08 07:27:50.393444 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:50.393413    2580 generic.go:356] "Generic (PLEG): container finished" podID="84e17bc5-2b17-4ac1-b96c-b328f7219f57" containerID="63bdae98e3090c682a587bd19b6ff86ced8f5dda91846a40778a895743da4c75" exitCode=0
Sep 08 07:27:50.393823 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:50.393504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7888-88659d" event={"ID":"84e17bc5-2b17-4ac1-b96c-b328f7219f57","Type":"ContainerDied","Data":"63bdae98e3090c682a587bd19b6ff86ced8f5dda91846a40778a895743da4c75"}
Sep 08 07:27:51.529595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:51.529569    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:51.555015 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:51.554987    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7888-88659d_84e17bc5-2b17-4ac1-b96c-b328f7219f57/mt-curl-7888-88659d/0.log"
Sep 08 07:27:51.575946 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:51.575852    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7888-88659d"]
Sep 08 07:27:51.577633 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:51.577608    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7888-88659d"]
Sep 08 07:27:51.672230 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:51.672136    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84mkr\" (UniqueName: \"kubernetes.io/projected/84e17bc5-2b17-4ac1-b96c-b328f7219f57-kube-api-access-84mkr\") pod \"84e17bc5-2b17-4ac1-b96c-b328f7219f57\" (UID: \"84e17bc5-2b17-4ac1-b96c-b328f7219f57\") "
Sep 08 07:27:51.674172 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:51.674145    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e17bc5-2b17-4ac1-b96c-b328f7219f57-kube-api-access-84mkr" (OuterVolumeSpecName: "kube-api-access-84mkr") pod "84e17bc5-2b17-4ac1-b96c-b328f7219f57" (UID: "84e17bc5-2b17-4ac1-b96c-b328f7219f57"). InnerVolumeSpecName "kube-api-access-84mkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:27:51.773866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:51.773816    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84mkr\" (UniqueName: \"kubernetes.io/projected/84e17bc5-2b17-4ac1-b96c-b328f7219f57-kube-api-access-84mkr\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:27:52.404961 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:52.404930    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:52.404961 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:52.404952    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5d80901e3f756a149f7b02f182080b17d3fd0938e4b7fa404e75e56e1d4a25c"
Sep 08 07:27:52.414352 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:27:52.414323    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7888-88659d\" is forbidden: User \"system:node:ip-10-0-139-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-109.ec2.internal' and this object" podUID="84e17bc5-2b17-4ac1-b96c-b328f7219f57" pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:52.849726 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:52.849698    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"]
Sep 08 07:27:52.866251 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:27:52.866215    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7888-88659d\" is forbidden: User \"system:node:ip-10-0-139-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-109.ec2.internal' and this object" podUID="84e17bc5-2b17-4ac1-b96c-b328f7219f57" pod="opendatahub/mt-curl-7888-88659d"
Sep 08 07:27:52.869439 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:52.869413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:52.931082 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:52.931051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:52.939945 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:52.939916    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84e17bc5-2b17-4ac1-b96c-b328f7219f57" path="/var/lib/kubelet/pods/84e17bc5-2b17-4ac1-b96c-b328f7219f57/volumes"
Sep 08 07:27:53.043786 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:53.043752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:53.145516 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:53.145408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:27:53.827689 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:53.827653    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"]
Sep 08 07:27:53.849042 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:27:53.849009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:03.575578 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:03.575546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:03.790802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:03.790770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:07.870218 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:07.870176    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl" podUID="b73db49b-6999-4938-964e-a02dcfd51652" containerName="main" containerID="cri-o://1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966" gracePeriod=45
Sep 08 07:28:08.135663 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.135600    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:28:08.227452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.227413    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-dshm\") pod \"b73db49b-6999-4938-964e-a02dcfd51652\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") "
Sep 08 07:28:08.227676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.227519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73db49b-6999-4938-964e-a02dcfd51652-tls-certs\") pod \"b73db49b-6999-4938-964e-a02dcfd51652\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") "
Sep 08 07:28:08.227676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.227548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-home\") pod \"b73db49b-6999-4938-964e-a02dcfd51652\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") "
Sep 08 07:28:08.227676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.227567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzkwv\" (UniqueName: \"kubernetes.io/projected/b73db49b-6999-4938-964e-a02dcfd51652-kube-api-access-jzkwv\") pod \"b73db49b-6999-4938-964e-a02dcfd51652\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") "
Sep 08 07:28:08.227856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.227721    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-tmp-dir\") pod \"b73db49b-6999-4938-964e-a02dcfd51652\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") "
Sep 08 07:28:08.227856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.227783    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-home" (OuterVolumeSpecName: "home") pod "b73db49b-6999-4938-964e-a02dcfd51652" (UID: "b73db49b-6999-4938-964e-a02dcfd51652"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:28:08.227856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.227837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-model-cache\") pod \"b73db49b-6999-4938-964e-a02dcfd51652\" (UID: \"b73db49b-6999-4938-964e-a02dcfd51652\") "
Sep 08 07:28:08.228011 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.227964    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b73db49b-6999-4938-964e-a02dcfd51652" (UID: "b73db49b-6999-4938-964e-a02dcfd51652"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:28:08.228112 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.228092    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-model-cache" (OuterVolumeSpecName: "model-cache") pod "b73db49b-6999-4938-964e-a02dcfd51652" (UID: "b73db49b-6999-4938-964e-a02dcfd51652"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:28:08.228228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.228191    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:08.228228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.228217    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:08.228335 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.228232    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:08.229901 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.229871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73db49b-6999-4938-964e-a02dcfd51652-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b73db49b-6999-4938-964e-a02dcfd51652" (UID: "b73db49b-6999-4938-964e-a02dcfd51652"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:28:08.230017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.229917    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-dshm" (OuterVolumeSpecName: "dshm") pod "b73db49b-6999-4938-964e-a02dcfd51652" (UID: "b73db49b-6999-4938-964e-a02dcfd51652"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:28:08.230017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.229925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73db49b-6999-4938-964e-a02dcfd51652-kube-api-access-jzkwv" (OuterVolumeSpecName: "kube-api-access-jzkwv") pod "b73db49b-6999-4938-964e-a02dcfd51652" (UID: "b73db49b-6999-4938-964e-a02dcfd51652"). InnerVolumeSpecName "kube-api-access-jzkwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:28:08.329396 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.329357    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b73db49b-6999-4938-964e-a02dcfd51652-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:08.329396 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.329389    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b73db49b-6999-4938-964e-a02dcfd51652-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:08.329396 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.329401    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzkwv\" (UniqueName: \"kubernetes.io/projected/b73db49b-6999-4938-964e-a02dcfd51652-kube-api-access-jzkwv\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:08.481185 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.481091    2580 generic.go:356] "Generic (PLEG): container finished" podID="b73db49b-6999-4938-964e-a02dcfd51652" containerID="1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966" exitCode=0
Sep 08 07:28:08.481185 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.481135    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"
Sep 08 07:28:08.481185 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.481175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl" event={"ID":"b73db49b-6999-4938-964e-a02dcfd51652","Type":"ContainerDied","Data":"1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966"}
Sep 08 07:28:08.481424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.481216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl" event={"ID":"b73db49b-6999-4938-964e-a02dcfd51652","Type":"ContainerDied","Data":"af565e0c4fa01d06441bb92c99fde1f09978900b3ea49838ce557dfd5fed04c3"}
Sep 08 07:28:08.481424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.481234    2580 scope.go:117] "RemoveContainer" containerID="1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966"
Sep 08 07:28:08.490229 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.490211    2580 scope.go:117] "RemoveContainer" containerID="1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966"
Sep 08 07:28:08.490530 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:28:08.490510    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966\": container with ID starting with 1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966 not found: ID does not exist" containerID="1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966"
Sep 08 07:28:08.490598 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.490538    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966"} err="failed to get container status \"1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966\": rpc error: code = NotFound desc = could not find container \"1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966\": container with ID starting with 1d41d951464655d2a6a0eb2139210f34f14852301161642dae93994214993966 not found: ID does not exist"
Sep 08 07:28:08.502038 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.502012    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"]
Sep 08 07:28:08.505987 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.505967    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/auth-test-model-fb59c2-kserve-6d8cf8df79-2hlrl"]
Sep 08 07:28:08.847513 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.847451    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5" podUID="0625353e-87c5-49c7-87ab-e258844c6b32" containerName="main" containerID="cri-o://9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93" gracePeriod=45
Sep 08 07:28:08.941299 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:08.941263    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b73db49b-6999-4938-964e-a02dcfd51652" path="/var/lib/kubelet/pods/b73db49b-6999-4938-964e-a02dcfd51652/volumes"
Sep 08 07:28:09.089240 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.089216    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:28:09.137187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-tmp-dir\") pod \"0625353e-87c5-49c7-87ab-e258844c6b32\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") "
Sep 08 07:28:09.137187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-model-cache\") pod \"0625353e-87c5-49c7-87ab-e258844c6b32\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") "
Sep 08 07:28:09.137408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137198    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-dshm\") pod \"0625353e-87c5-49c7-87ab-e258844c6b32\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") "
Sep 08 07:28:09.137408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngdlr\" (UniqueName: \"kubernetes.io/projected/0625353e-87c5-49c7-87ab-e258844c6b32-kube-api-access-ngdlr\") pod \"0625353e-87c5-49c7-87ab-e258844c6b32\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") "
Sep 08 07:28:09.137408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137293    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0625353e-87c5-49c7-87ab-e258844c6b32" (UID: "0625353e-87c5-49c7-87ab-e258844c6b32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:28:09.137408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-home\") pod \"0625353e-87c5-49c7-87ab-e258844c6b32\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") "
Sep 08 07:28:09.137408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0625353e-87c5-49c7-87ab-e258844c6b32-tls-certs\") pod \"0625353e-87c5-49c7-87ab-e258844c6b32\" (UID: \"0625353e-87c5-49c7-87ab-e258844c6b32\") "
Sep 08 07:28:09.137694 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-model-cache" (OuterVolumeSpecName: "model-cache") pod "0625353e-87c5-49c7-87ab-e258844c6b32" (UID: "0625353e-87c5-49c7-87ab-e258844c6b32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:28:09.137694 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137592    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-home" (OuterVolumeSpecName: "home") pod "0625353e-87c5-49c7-87ab-e258844c6b32" (UID: "0625353e-87c5-49c7-87ab-e258844c6b32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:28:09.137787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137728    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:09.137787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137742    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:09.137787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.137753    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:09.139509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.139487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-dshm" (OuterVolumeSpecName: "dshm") pod "0625353e-87c5-49c7-87ab-e258844c6b32" (UID: "0625353e-87c5-49c7-87ab-e258844c6b32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:28:09.139593 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.139486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0625353e-87c5-49c7-87ab-e258844c6b32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0625353e-87c5-49c7-87ab-e258844c6b32" (UID: "0625353e-87c5-49c7-87ab-e258844c6b32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:28:09.139593 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.139573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0625353e-87c5-49c7-87ab-e258844c6b32-kube-api-access-ngdlr" (OuterVolumeSpecName: "kube-api-access-ngdlr") pod "0625353e-87c5-49c7-87ab-e258844c6b32" (UID: "0625353e-87c5-49c7-87ab-e258844c6b32"). InnerVolumeSpecName "kube-api-access-ngdlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:28:09.238399 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.238347    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngdlr\" (UniqueName: \"kubernetes.io/projected/0625353e-87c5-49c7-87ab-e258844c6b32-kube-api-access-ngdlr\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:09.238399 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.238394    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0625353e-87c5-49c7-87ab-e258844c6b32-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:09.238399 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.238406    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0625353e-87c5-49c7-87ab-e258844c6b32-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:28:09.487871 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.487779    2580 generic.go:356] "Generic (PLEG): container finished" podID="0625353e-87c5-49c7-87ab-e258844c6b32" containerID="9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93" exitCode=0
Sep 08 07:28:09.487871 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.487845    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"
Sep 08 07:28:09.488095 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.487865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5" event={"ID":"0625353e-87c5-49c7-87ab-e258844c6b32","Type":"ContainerDied","Data":"9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93"}
Sep 08 07:28:09.488095 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.487911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5" event={"ID":"0625353e-87c5-49c7-87ab-e258844c6b32","Type":"ContainerDied","Data":"1dddac1d736008e2fb96b6fb0acbad1c9d67af33c55d1f295246f5714225e937"}
Sep 08 07:28:09.488095 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.487932    2580 scope.go:117] "RemoveContainer" containerID="9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93"
Sep 08 07:28:09.497621 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.497603    2580 scope.go:117] "RemoveContainer" containerID="9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93"
Sep 08 07:28:09.497898 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:28:09.497880    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93\": container with ID starting with 9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93 not found: ID does not exist" containerID="9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93"
Sep 08 07:28:09.497968 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.497905    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93"} err="failed to get container status \"9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93\": rpc error: code = NotFound desc = could not find container \"9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93\": container with ID starting with 9b10beec205df741ff4cc29cc057147d44a6c52e606dbff3d2d1da67dda91f93 not found: ID does not exist"
Sep 08 07:28:09.509533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.509508    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"]
Sep 08 07:28:09.515136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:09.515116    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/auth-test-model-217bc2-kserve-6dc7796fb7-crrl5"]
Sep 08 07:28:10.941253 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:10.941218    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0625353e-87c5-49c7-87ab-e258844c6b32" path="/var/lib/kubelet/pods/0625353e-87c5-49c7-87ab-e258844c6b32/volumes"
Sep 08 07:28:16.495239 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.495203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:16.595836 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.595801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:16.651870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.651833    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"]
Sep 08 07:28:16.652379 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652362    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0625353e-87c5-49c7-87ab-e258844c6b32" containerName="main"
Sep 08 07:28:16.652428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652383    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0625353e-87c5-49c7-87ab-e258844c6b32" containerName="main"
Sep 08 07:28:16.652428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652400    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b73db49b-6999-4938-964e-a02dcfd51652" containerName="main"
Sep 08 07:28:16.652428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652408    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b73db49b-6999-4938-964e-a02dcfd51652" containerName="main"
Sep 08 07:28:16.652428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652422    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e17bc5-2b17-4ac1-b96c-b328f7219f57" containerName="mt-curl-7888-88659d"
Sep 08 07:28:16.652566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652432    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e17bc5-2b17-4ac1-b96c-b328f7219f57" containerName="mt-curl-7888-88659d"
Sep 08 07:28:16.652566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652541    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0625353e-87c5-49c7-87ab-e258844c6b32" containerName="main"
Sep 08 07:28:16.652566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652553    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e17bc5-2b17-4ac1-b96c-b328f7219f57" containerName="mt-curl-7888-88659d"
Sep 08 07:28:16.652566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.652565    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b73db49b-6999-4938-964e-a02dcfd51652" containerName="main"
Sep 08 07:28:16.655898 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.655880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.658697 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.658666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"sub-test-model-fb59c2-kserve-self-signed-certs\""
Sep 08 07:28:16.658809 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.658724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"openshift-service-ca.crt\""
Sep 08 07:28:16.658809 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.658764    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"default-dockercfg-ltj4x\""
Sep 08 07:28:16.659655 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.659631    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"kube-root-ca.crt\""
Sep 08 07:28:16.662536 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.662512    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"]
Sep 08 07:28:16.672303 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.672271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:16.814150 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.814055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3eebf1-3f76-4470-8580-810716963701-tls-certs\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.814150 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.814135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8mw\" (UniqueName: \"kubernetes.io/projected/2f3eebf1-3f76-4470-8580-810716963701-kube-api-access-fz8mw\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.814352 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.814169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-model-cache\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.814352 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.814193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-home\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.814352 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.814266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-dshm\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.814352 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.814303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-tmp-dir\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915186 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-dshm\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915186 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-tmp-dir\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3eebf1-3f76-4470-8580-810716963701-tls-certs\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8mw\" (UniqueName: \"kubernetes.io/projected/2f3eebf1-3f76-4470-8580-810716963701-kube-api-access-fz8mw\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-model-cache\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-home\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-tmp-dir\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-home\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.915772 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.915743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-model-cache\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.917441 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.917421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-dshm\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.917860 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.917839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3eebf1-3f76-4470-8580-810716963701-tls-certs\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.922801 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.922780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8mw\" (UniqueName: \"kubernetes.io/projected/2f3eebf1-3f76-4470-8580-810716963701-kube-api-access-fz8mw\") pod \"sub-test-model-fb59c2-kserve-6557965c49-95b5c\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:16.965913 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:16.965879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:17.235223 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:17.235198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"]
Sep 08 07:28:17.236935 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:28:17.236897    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f3eebf1_3f76_4470_8580_810716963701.slice/crio-da95ab531c655d45d4429278b432151d5c5cb9e671a70ae49502a4f93b8a3b42 WatchSource:0}: Error finding container da95ab531c655d45d4429278b432151d5c5cb9e671a70ae49502a4f93b8a3b42: Status 404 returned error can't find the container with id da95ab531c655d45d4429278b432151d5c5cb9e671a70ae49502a4f93b8a3b42
Sep 08 07:28:17.531912 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:17.531820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c" event={"ID":"2f3eebf1-3f76-4470-8580-810716963701","Type":"ContainerStarted","Data":"bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a"}
Sep 08 07:28:17.531912 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:17.531859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c" event={"ID":"2f3eebf1-3f76-4470-8580-810716963701","Type":"ContainerStarted","Data":"da95ab531c655d45d4429278b432151d5c5cb9e671a70ae49502a4f93b8a3b42"}
Sep 08 07:28:17.548892 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:17.548838    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c" podStartSLOduration=1.548823933 podStartE2EDuration="1.548823933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:28:17.547448473 +0000 UTC m=+1413.230908627" watchObservedRunningTime="2026-09-08 07:28:17.548823933 +0000 UTC m=+1413.232284140"
Sep 08 07:28:24.686214 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:24.686177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:26.966063 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:26.966023    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:26.966424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:26.966081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:26.978747 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:26.978720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:27.591895 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:27.591861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:28:30.609856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:30.609821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:30.682878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:30.682846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:30.935833 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:30.935798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:31.222480 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:31.222383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:31.444633 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:31.444600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:35.480943 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:35.480908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:35.579159 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:35.579121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:35.888182 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:35.888127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:35.990582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:35.990549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:36.176308 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.176227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:36.496901 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.496818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:36.523757 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.523730    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"]
Sep 08 07:28:36.529262 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.529240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:36.531427 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.531406    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7e01ea\""
Sep 08 07:28:36.531555 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.531449    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7e01ea-dockercfg-vh5c9\""
Sep 08 07:28:36.539592 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.539567    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"]
Sep 08 07:28:36.546047 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.546026    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"]
Sep 08 07:28:36.549642 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.549626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:36.560503 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.560481    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"]
Sep 08 07:28:36.590874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.590852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:36.604102 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.604075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ddacea-0be8-4e99-96ac-6394bf85052d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp\" (UID: \"a1ddacea-0be8-4e99-96ac-6394bf85052d\") " pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:36.604204 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.604133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv\" (UID: \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:36.604204 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.604177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk4gz\" (UniqueName: \"kubernetes.io/projected/a1ddacea-0be8-4e99-96ac-6394bf85052d-kube-api-access-qk4gz\") pod \"payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp\" (UID: \"a1ddacea-0be8-4e99-96ac-6394bf85052d\") " pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:36.604318 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.604251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lv7f\" (UniqueName: \"kubernetes.io/projected/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-kube-api-access-5lv7f\") pod \"payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv\" (UID: \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:36.705883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.705849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ddacea-0be8-4e99-96ac-6394bf85052d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp\" (UID: \"a1ddacea-0be8-4e99-96ac-6394bf85052d\") " pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:36.706087 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.705897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv\" (UID: \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:36.706087 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.705915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4gz\" (UniqueName: \"kubernetes.io/projected/a1ddacea-0be8-4e99-96ac-6394bf85052d-kube-api-access-qk4gz\") pod \"payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp\" (UID: \"a1ddacea-0be8-4e99-96ac-6394bf85052d\") " pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:36.706087 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.705980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lv7f\" (UniqueName: \"kubernetes.io/projected/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-kube-api-access-5lv7f\") pod \"payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv\" (UID: \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:36.706698 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.706675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ddacea-0be8-4e99-96ac-6394bf85052d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp\" (UID: \"a1ddacea-0be8-4e99-96ac-6394bf85052d\") " pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:36.706698 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.706683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv\" (UID: \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:36.717509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.717447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4gz\" (UniqueName: \"kubernetes.io/projected/a1ddacea-0be8-4e99-96ac-6394bf85052d-kube-api-access-qk4gz\") pod \"payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp\" (UID: \"a1ddacea-0be8-4e99-96ac-6394bf85052d\") " pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:36.717718 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.717700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lv7f\" (UniqueName: \"kubernetes.io/projected/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-kube-api-access-5lv7f\") pod \"payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv\" (UID: \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:36.840606 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.840577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:36.860208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:36.860185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:37.114026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.114000    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"]
Sep 08 07:28:37.116125 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:28:37.116099    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d293b7_78dc_4edb_b1d1_e0ab2ec0d0a6.slice/crio-120395d095430824a328726e487215a363bdeefaa9ed34f5c23a22afa2e84494 WatchSource:0}: Error finding container 120395d095430824a328726e487215a363bdeefaa9ed34f5c23a22afa2e84494: Status 404 returned error can't find the container with id 120395d095430824a328726e487215a363bdeefaa9ed34f5c23a22afa2e84494
Sep 08 07:28:37.143391 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.143371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"]
Sep 08 07:28:37.153850 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:28:37.153818    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ddacea_0be8_4e99_96ac_6394bf85052d.slice/crio-52b66e9024b2183d70f898eb654fea854d8d05c9b810ce432dca6bf0d3c62e8f WatchSource:0}: Error finding container 52b66e9024b2183d70f898eb654fea854d8d05c9b810ce432dca6bf0d3c62e8f: Status 404 returned error can't find the container with id 52b66e9024b2183d70f898eb654fea854d8d05c9b810ce432dca6bf0d3c62e8f
Sep 08 07:28:37.636880 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.636837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp" event={"ID":"a1ddacea-0be8-4e99-96ac-6394bf85052d","Type":"ContainerStarted","Data":"c6f4525df68aa640cb1cc9468c87a96afa8c2e86a85c81b8297aaee77286809e"}
Sep 08 07:28:37.637360 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.636901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp" event={"ID":"a1ddacea-0be8-4e99-96ac-6394bf85052d","Type":"ContainerStarted","Data":"52b66e9024b2183d70f898eb654fea854d8d05c9b810ce432dca6bf0d3c62e8f"}
Sep 08 07:28:37.637360 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.636931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:37.638863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.638837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv" event={"ID":"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6","Type":"ContainerStarted","Data":"dd8afeb4b8f03a71028a225bd01d65c00d0456a5e9503803e6d313b00d5b26c6"}
Sep 08 07:28:37.639010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.638889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:37.639010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.638912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv" event={"ID":"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6","Type":"ContainerStarted","Data":"120395d095430824a328726e487215a363bdeefaa9ed34f5c23a22afa2e84494"}
Sep 08 07:28:37.655178 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.655124    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp" podStartSLOduration=1.6551023809999998 podStartE2EDuration="1.655102381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:28:37.653486619 +0000 UTC m=+1433.336946773" watchObservedRunningTime="2026-09-08 07:28:37.655102381 +0000 UTC m=+1433.338562536"
Sep 08 07:28:37.671251 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:37.671203    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv" podStartSLOduration=1.671188082 podStartE2EDuration="1.671188082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:28:37.669306973 +0000 UTC m=+1433.352767129" watchObservedRunningTime="2026-09-08 07:28:37.671188082 +0000 UTC m=+1433.354648235"
Sep 08 07:28:44.724574 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:44.724538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:44.848313 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:44.848281    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:48.236433 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.236401    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"]
Sep 08 07:28:48.241143 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.241122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.243735 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.243709    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"openshift-service-ca.crt\""
Sep 08 07:28:48.243844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.243737    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"kube-root-ca.crt\""
Sep 08 07:28:48.243844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.243709    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"default-dockercfg-r9gnf\""
Sep 08 07:28:48.243844 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.243784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"sub-test-model-217bc2-kserve-self-signed-certs\""
Sep 08 07:28:48.248019 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.247998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"]
Sep 08 07:28:48.320334 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.320301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-model-cache\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.320551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.320351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-dshm\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.320551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.320382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-home\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.320551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.320440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11524b31-d7e2-4d9a-90b9-e57929f8248d-tls-certs\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.320551 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.320517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-tmp-dir\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.320762 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.320591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbkp\" (UniqueName: \"kubernetes.io/projected/11524b31-d7e2-4d9a-90b9-e57929f8248d-kube-api-access-scbkp\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.421658 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.421626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbkp\" (UniqueName: \"kubernetes.io/projected/11524b31-d7e2-4d9a-90b9-e57929f8248d-kube-api-access-scbkp\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.421840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.421687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-model-cache\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.421840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.421728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-dshm\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.421840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.421771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-home\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.421840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.421801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11524b31-d7e2-4d9a-90b9-e57929f8248d-tls-certs\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.421840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.421835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-tmp-dir\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.422209 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.422186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-model-cache\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.422275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.422223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-home\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.422312 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.422273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-tmp-dir\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.424019 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.424002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-dshm\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.424202 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.424186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11524b31-d7e2-4d9a-90b9-e57929f8248d-tls-certs\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.432384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.432361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbkp\" (UniqueName: \"kubernetes.io/projected/11524b31-d7e2-4d9a-90b9-e57929f8248d-kube-api-access-scbkp\") pod \"sub-test-model-217bc2-kserve-5b9974dd57-7hbrz\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.433537 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.433517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:48.552007 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.551899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:48.645516 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.645486    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:28:48.645726 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.645571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:28:48.824745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:48.824717    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"]
Sep 08 07:28:48.826891 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:28:48.826856    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11524b31_d7e2_4d9a_90b9_e57929f8248d.slice/crio-d510ef4e02e9014559f7081162b413ed912b0727510a9ece76447886591c3c94 WatchSource:0}: Error finding container d510ef4e02e9014559f7081162b413ed912b0727510a9ece76447886591c3c94: Status 404 returned error can't find the container with id d510ef4e02e9014559f7081162b413ed912b0727510a9ece76447886591c3c94
Sep 08 07:28:49.701328 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:49.701293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz" event={"ID":"11524b31-d7e2-4d9a-90b9-e57929f8248d","Type":"ContainerStarted","Data":"18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1"}
Sep 08 07:28:49.701737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:49.701338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz" event={"ID":"11524b31-d7e2-4d9a-90b9-e57929f8248d","Type":"ContainerStarted","Data":"d510ef4e02e9014559f7081162b413ed912b0727510a9ece76447886591c3c94"}
Sep 08 07:28:49.717684 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:49.717614    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz" podStartSLOduration=1.717595617 podStartE2EDuration="1.717595617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:28:49.716629079 +0000 UTC m=+1445.400089233" watchObservedRunningTime="2026-09-08 07:28:49.717595617 +0000 UTC m=+1445.401055770"
Sep 08 07:28:58.552615 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:58.552575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:58.553028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:58.552627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:58.565613 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:58.565580    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:58.757176 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:58.757147    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:28:59.472242 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:59.472213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:28:59.902646 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:28:59.902601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:00.182756 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:00.182650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:00.477129 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:00.477020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:03.703069 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:03.703037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:03.802866 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:03.802813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:04.882857 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:04.882822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:04.983312 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:04.983276    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:05.262577 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:05.262449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:05.361751 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:05.361714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:05.465864 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:05.465827    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:20.498870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:20.498830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:22.347952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:22.347917    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:25.871974 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:25.871937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:28.621482 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:28.621425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:31.340149 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.340113    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"]
Sep 08 07:29:31.343727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.343708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"
Sep 08 07:29:31.345679 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.345656    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:29:31.345903 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.345887    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:29:31.351068 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.351040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"]
Sep 08 07:29:31.441363 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.441336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43542b5-af13-4329-b6c7-3c8afa10481e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp\" (UID: \"e43542b5-af13-4329-b6c7-3c8afa10481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"
Sep 08 07:29:31.541960 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.541927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43542b5-af13-4329-b6c7-3c8afa10481e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp\" (UID: \"e43542b5-af13-4329-b6c7-3c8afa10481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"
Sep 08 07:29:31.542547 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.542527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43542b5-af13-4329-b6c7-3c8afa10481e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp\" (UID: \"e43542b5-af13-4329-b6c7-3c8afa10481e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"
Sep 08 07:29:31.632705 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.632618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:31.657344 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.657309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"
Sep 08 07:29:31.931529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:31.931497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"]
Sep 08 07:29:31.934342 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:29:31.934313    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43542b5_af13_4329_b6c7_3c8afa10481e.slice/crio-0917623c258120ade410ee6c80eb35b64ce31826f353cdffe537581dfc113424 WatchSource:0}: Error finding container 0917623c258120ade410ee6c80eb35b64ce31826f353cdffe537581dfc113424: Status 404 returned error can't find the container with id 0917623c258120ade410ee6c80eb35b64ce31826f353cdffe537581dfc113424
Sep 08 07:29:32.924528 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:32.924418    2580 generic.go:356] "Generic (PLEG): container finished" podID="e43542b5-af13-4329-b6c7-3c8afa10481e" containerID="d46e2f0e7190f510ceae95cbff8c1bac3f12a89d5811c62b07461c05a5cd56db" exitCode=0
Sep 08 07:29:32.924528 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:32.924493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp" event={"ID":"e43542b5-af13-4329-b6c7-3c8afa10481e","Type":"ContainerDied","Data":"d46e2f0e7190f510ceae95cbff8c1bac3f12a89d5811c62b07461c05a5cd56db"}
Sep 08 07:29:32.924913 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:32.924534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp" event={"ID":"e43542b5-af13-4329-b6c7-3c8afa10481e","Type":"ContainerStarted","Data":"0917623c258120ade410ee6c80eb35b64ce31826f353cdffe537581dfc113424"}
Sep 08 07:29:34.061999 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:34.061979    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"
Sep 08 07:29:34.168210 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:34.168174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43542b5-af13-4329-b6c7-3c8afa10481e-maas-api-service-ca\") pod \"e43542b5-af13-4329-b6c7-3c8afa10481e\" (UID: \"e43542b5-af13-4329-b6c7-3c8afa10481e\") "
Sep 08 07:29:34.168541 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:34.168517    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43542b5-af13-4329-b6c7-3c8afa10481e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e43542b5-af13-4329-b6c7-3c8afa10481e" (UID: "e43542b5-af13-4329-b6c7-3c8afa10481e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:29:34.269221 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:34.269137    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43542b5-af13-4329-b6c7-3c8afa10481e-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:29:34.937372 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:34.937344    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"
Sep 08 07:29:34.940767 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:34.940741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp" event={"ID":"e43542b5-af13-4329-b6c7-3c8afa10481e","Type":"ContainerDied","Data":"0917623c258120ade410ee6c80eb35b64ce31826f353cdffe537581dfc113424"}
Sep 08 07:29:34.940890 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:34.940782    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0917623c258120ade410ee6c80eb35b64ce31826f353cdffe537581dfc113424"
Sep 08 07:29:43.493341 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.493298    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7888-e5ed82"]
Sep 08 07:29:43.493828 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.493737    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e43542b5-af13-4329-b6c7-3c8afa10481e" containerName="revoke-keys"
Sep 08 07:29:43.493828 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.493749    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e43542b5-af13-4329-b6c7-3c8afa10481e" containerName="revoke-keys"
Sep 08 07:29:43.493828 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.493826    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e43542b5-af13-4329-b6c7-3c8afa10481e" containerName="revoke-keys"
Sep 08 07:29:43.497256 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.497238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7888-e5ed82"
Sep 08 07:29:43.499453 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.499424    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lxxzx\""
Sep 08 07:29:43.500950 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.500923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7888-e5ed82"]
Sep 08 07:29:43.560897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.560863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpzt\" (UniqueName: \"kubernetes.io/projected/285335af-4e95-494e-a0bf-80d7e2e2670a-kube-api-access-qvpzt\") pod \"mt-curl-7888-e5ed82\" (UID: \"285335af-4e95-494e-a0bf-80d7e2e2670a\") " pod="opendatahub/mt-curl-7888-e5ed82"
Sep 08 07:29:43.661574 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.661533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpzt\" (UniqueName: \"kubernetes.io/projected/285335af-4e95-494e-a0bf-80d7e2e2670a-kube-api-access-qvpzt\") pod \"mt-curl-7888-e5ed82\" (UID: \"285335af-4e95-494e-a0bf-80d7e2e2670a\") " pod="opendatahub/mt-curl-7888-e5ed82"
Sep 08 07:29:43.669998 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.669960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpzt\" (UniqueName: \"kubernetes.io/projected/285335af-4e95-494e-a0bf-80d7e2e2670a-kube-api-access-qvpzt\") pod \"mt-curl-7888-e5ed82\" (UID: \"285335af-4e95-494e-a0bf-80d7e2e2670a\") " pod="opendatahub/mt-curl-7888-e5ed82"
Sep 08 07:29:43.807343 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:43.807253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7888-e5ed82"
Sep 08 07:29:44.080230 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:44.080204    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7888-e5ed82"]
Sep 08 07:29:44.082773 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:29:44.082744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod285335af_4e95_494e_a0bf_80d7e2e2670a.slice/crio-5346e0f5b7654d718305eb67e9015a549e29ad5fb21b8f2188f0db96d70a2b99 WatchSource:0}: Error finding container 5346e0f5b7654d718305eb67e9015a549e29ad5fb21b8f2188f0db96d70a2b99: Status 404 returned error can't find the container with id 5346e0f5b7654d718305eb67e9015a549e29ad5fb21b8f2188f0db96d70a2b99
Sep 08 07:29:44.989122 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:44.989043    2580 generic.go:356] "Generic (PLEG): container finished" podID="285335af-4e95-494e-a0bf-80d7e2e2670a" containerID="68038fca2ffe217b162ed7978c0e9fb816c0bb5bf3e99a31db2602c0ce771bf8" exitCode=0
Sep 08 07:29:44.989487 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:44.989126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7888-e5ed82" event={"ID":"285335af-4e95-494e-a0bf-80d7e2e2670a","Type":"ContainerDied","Data":"68038fca2ffe217b162ed7978c0e9fb816c0bb5bf3e99a31db2602c0ce771bf8"}
Sep 08 07:29:44.989487 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:44.989169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7888-e5ed82" event={"ID":"285335af-4e95-494e-a0bf-80d7e2e2670a","Type":"ContainerStarted","Data":"5346e0f5b7654d718305eb67e9015a549e29ad5fb21b8f2188f0db96d70a2b99"}
Sep 08 07:29:45.041993 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:45.041964    2580 scope.go:117] "RemoveContainer" containerID="fbd6e135977c388b0e034f22154508118497229e26e3b2e66803f219ddf2c612"
Sep 08 07:29:46.125802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.125775    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7888-e5ed82"
Sep 08 07:29:46.144880 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.144849    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7888-e5ed82_285335af-4e95-494e-a0bf-80d7e2e2670a/mt-curl-7888-e5ed82/0.log"
Sep 08 07:29:46.165806 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.165582    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7888-e5ed82"]
Sep 08 07:29:46.168759 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.168727    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7888-e5ed82"]
Sep 08 07:29:46.185020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.184996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvpzt\" (UniqueName: \"kubernetes.io/projected/285335af-4e95-494e-a0bf-80d7e2e2670a-kube-api-access-qvpzt\") pod \"285335af-4e95-494e-a0bf-80d7e2e2670a\" (UID: \"285335af-4e95-494e-a0bf-80d7e2e2670a\") "
Sep 08 07:29:46.187070 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.187037    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285335af-4e95-494e-a0bf-80d7e2e2670a-kube-api-access-qvpzt" (OuterVolumeSpecName: "kube-api-access-qvpzt") pod "285335af-4e95-494e-a0bf-80d7e2e2670a" (UID: "285335af-4e95-494e-a0bf-80d7e2e2670a"). InnerVolumeSpecName "kube-api-access-qvpzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:29:46.286859 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.286769    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvpzt\" (UniqueName: \"kubernetes.io/projected/285335af-4e95-494e-a0bf-80d7e2e2670a-kube-api-access-qvpzt\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:29:46.824578 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.824539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:46.859895 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.859865    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"]
Sep 08 07:29:46.860165 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.860137    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp" podUID="a1ddacea-0be8-4e99-96ac-6394bf85052d" containerName="payload-processing" containerID="cri-o://c6f4525df68aa640cb1cc9468c87a96afa8c2e86a85c81b8297aaee77286809e" gracePeriod=30
Sep 08 07:29:46.865523 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.865492    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"]
Sep 08 07:29:46.865749 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.865724    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv" podUID="92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6" containerName="payload-pre-processing" containerID="cri-o://dd8afeb4b8f03a71028a225bd01d65c00d0456a5e9503803e6d313b00d5b26c6" gracePeriod=30
Sep 08 07:29:46.909657 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.909622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:46.941071 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:46.941039    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285335af-4e95-494e-a0bf-80d7e2e2670a" path="/var/lib/kubelet/pods/285335af-4e95-494e-a0bf-80d7e2e2670a/volumes"
Sep 08 07:29:47.004486 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.003743    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7888-e5ed82"
Sep 08 07:29:47.004486 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.003775    2580 scope.go:117] "RemoveContainer" containerID="68038fca2ffe217b162ed7978c0e9fb816c0bb5bf3e99a31db2602c0ce771bf8"
Sep 08 07:29:47.010202 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.010152    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1ddacea-0be8-4e99-96ac-6394bf85052d" containerID="c6f4525df68aa640cb1cc9468c87a96afa8c2e86a85c81b8297aaee77286809e" exitCode=0
Sep 08 07:29:47.010337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.010226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp" event={"ID":"a1ddacea-0be8-4e99-96ac-6394bf85052d","Type":"ContainerDied","Data":"c6f4525df68aa640cb1cc9468c87a96afa8c2e86a85c81b8297aaee77286809e"}
Sep 08 07:29:47.011193 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.011138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:47.012315 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.012290    2580 generic.go:356] "Generic (PLEG): container finished" podID="92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6" containerID="dd8afeb4b8f03a71028a225bd01d65c00d0456a5e9503803e6d313b00d5b26c6" exitCode=0
Sep 08 07:29:47.012418 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.012385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv" event={"ID":"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6","Type":"ContainerDied","Data":"dd8afeb4b8f03a71028a225bd01d65c00d0456a5e9503803e6d313b00d5b26c6"}
Sep 08 07:29:47.042262 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.042206    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"]
Sep 08 07:29:47.043862 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.043801    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7e01ea-gjmzp"]
Sep 08 07:29:47.111527 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.111436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:47.150687 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.150666    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:29:47.152929 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.152911    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:29:47.197641 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.197617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lv7f\" (UniqueName: \"kubernetes.io/projected/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-kube-api-access-5lv7f\") pod \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\" (UID: \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\") "
Sep 08 07:29:47.197789 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.197662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk4gz\" (UniqueName: \"kubernetes.io/projected/a1ddacea-0be8-4e99-96ac-6394bf85052d-kube-api-access-qk4gz\") pod \"a1ddacea-0be8-4e99-96ac-6394bf85052d\" (UID: \"a1ddacea-0be8-4e99-96ac-6394bf85052d\") "
Sep 08 07:29:47.197789 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.197703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-plugins-config-volume\") pod \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\" (UID: \"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6\") "
Sep 08 07:29:47.197789 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.197756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ddacea-0be8-4e99-96ac-6394bf85052d-plugins-config-volume\") pod \"a1ddacea-0be8-4e99-96ac-6394bf85052d\" (UID: \"a1ddacea-0be8-4e99-96ac-6394bf85052d\") "
Sep 08 07:29:47.198149 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.198121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ddacea-0be8-4e99-96ac-6394bf85052d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1ddacea-0be8-4e99-96ac-6394bf85052d" (UID: "a1ddacea-0be8-4e99-96ac-6394bf85052d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:29:47.198149 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.198132    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6" (UID: "92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:29:47.199761 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.199737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-kube-api-access-5lv7f" (OuterVolumeSpecName: "kube-api-access-5lv7f") pod "92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6" (UID: "92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6"). InnerVolumeSpecName "kube-api-access-5lv7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:29:47.199849 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.199801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ddacea-0be8-4e99-96ac-6394bf85052d-kube-api-access-qk4gz" (OuterVolumeSpecName: "kube-api-access-qk4gz") pod "a1ddacea-0be8-4e99-96ac-6394bf85052d" (UID: "a1ddacea-0be8-4e99-96ac-6394bf85052d"). InnerVolumeSpecName "kube-api-access-qk4gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:29:47.298586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.298549    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:29:47.298586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.298580    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ddacea-0be8-4e99-96ac-6394bf85052d-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:29:47.298586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.298591    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lv7f\" (UniqueName: \"kubernetes.io/projected/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6-kube-api-access-5lv7f\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:29:47.298824 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.298599    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk4gz\" (UniqueName: \"kubernetes.io/projected/a1ddacea-0be8-4e99-96ac-6394bf85052d-kube-api-access-qk4gz\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:29:47.370951 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.370864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:47.782184 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:47.782103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:48.020151 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.020124    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"
Sep 08 07:29:48.020151 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.020154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp" event={"ID":"a1ddacea-0be8-4e99-96ac-6394bf85052d","Type":"ContainerDied","Data":"52b66e9024b2183d70f898eb654fea854d8d05c9b810ce432dca6bf0d3c62e8f"}
Sep 08 07:29:48.020440 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.020222    2580 scope.go:117] "RemoveContainer" containerID="c6f4525df68aa640cb1cc9468c87a96afa8c2e86a85c81b8297aaee77286809e"
Sep 08 07:29:48.021561 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.021539    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"
Sep 08 07:29:48.021656 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.021568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv" event={"ID":"92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6","Type":"ContainerDied","Data":"120395d095430824a328726e487215a363bdeefaa9ed34f5c23a22afa2e84494"}
Sep 08 07:29:48.030400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.030282    2580 scope.go:117] "RemoveContainer" containerID="dd8afeb4b8f03a71028a225bd01d65c00d0456a5e9503803e6d313b00d5b26c6"
Sep 08 07:29:48.046222 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.046196    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"]
Sep 08 07:29:48.052237 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.052213    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7e01ea-59598d5789-bxmhp"]
Sep 08 07:29:48.062330 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.062306    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"]
Sep 08 07:29:48.066316 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.066293    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7e01ea-749d4b5d4b-hjnlv"]
Sep 08 07:29:48.941264 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.941230    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6" path="/var/lib/kubelet/pods/92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6/volumes"
Sep 08 07:29:48.941744 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.941611    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1ddacea-0be8-4e99-96ac-6394bf85052d" path="/var/lib/kubelet/pods/a1ddacea-0be8-4e99-96ac-6394bf85052d/volumes"
Sep 08 07:29:48.941907 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:48.941894    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e43542b5-af13-4329-b6c7-3c8afa10481e" path="/var/lib/kubelet/pods/e43542b5-af13-4329-b6c7-3c8afa10481e/volumes"
Sep 08 07:29:50.563482 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:50.563421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:51.729067 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:51.729034    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"]
Sep 08 07:29:51.811751 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:51.811712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:52.633816 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:52.633782    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"]
Sep 08 07:29:52.651987 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:52.651952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:53.352261 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.352229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:53.418435 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.418393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:53.641530 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.641426    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:53.912361 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912278    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"]
Sep 08 07:29:53.912694 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912680    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6" containerName="payload-pre-processing"
Sep 08 07:29:53.912745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912695    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6" containerName="payload-pre-processing"
Sep 08 07:29:53.912745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912713    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285335af-4e95-494e-a0bf-80d7e2e2670a" containerName="mt-curl-7888-e5ed82"
Sep 08 07:29:53.912745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912719    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="285335af-4e95-494e-a0bf-80d7e2e2670a" containerName="mt-curl-7888-e5ed82"
Sep 08 07:29:53.912745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912734    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ddacea-0be8-4e99-96ac-6394bf85052d" containerName="payload-processing"
Sep 08 07:29:53.912745 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912742    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ddacea-0be8-4e99-96ac-6394bf85052d" containerName="payload-processing"
Sep 08 07:29:53.912905 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912809    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ddacea-0be8-4e99-96ac-6394bf85052d" containerName="payload-processing"
Sep 08 07:29:53.912905 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912819    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d293b7-78dc-4edb-b1d1-e0ab2ec0d0a6" containerName="payload-pre-processing"
Sep 08 07:29:53.912905 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.912827    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="285335af-4e95-494e-a0bf-80d7e2e2670a" containerName="mt-curl-7888-e5ed82"
Sep 08 07:29:53.917555 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.917538    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"
Sep 08 07:29:53.919308 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.919287    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:29:53.919424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.919410    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:29:53.922641 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.922615    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"]
Sep 08 07:29:53.960484 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:53.960432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7362bc-2b5a-408e-9a09-b9ef28888d43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99\" (UID: \"8d7362bc-2b5a-408e-9a09-b9ef28888d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"
Sep 08 07:29:54.061775 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:54.061739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7362bc-2b5a-408e-9a09-b9ef28888d43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99\" (UID: \"8d7362bc-2b5a-408e-9a09-b9ef28888d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"
Sep 08 07:29:54.062401 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:54.062382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7362bc-2b5a-408e-9a09-b9ef28888d43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99\" (UID: \"8d7362bc-2b5a-408e-9a09-b9ef28888d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"
Sep 08 07:29:54.074495 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:54.074449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:29:54.229836 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:54.229740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"
Sep 08 07:29:54.500727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:54.500651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"]
Sep 08 07:29:54.504080 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:29:54.504051    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7362bc_2b5a_408e_9a09_b9ef28888d43.slice/crio-c4f3f85eb149c32d6f570bfcc635003600a05e83b0fc5e11cec57689f046a4a8 WatchSource:0}: Error finding container c4f3f85eb149c32d6f570bfcc635003600a05e83b0fc5e11cec57689f046a4a8: Status 404 returned error can't find the container with id c4f3f85eb149c32d6f570bfcc635003600a05e83b0fc5e11cec57689f046a4a8
Sep 08 07:29:55.059940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:55.059907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99" event={"ID":"8d7362bc-2b5a-408e-9a09-b9ef28888d43","Type":"ContainerStarted","Data":"bd5f7bb3932313cfe0afa0a139eb1ce62d527ba2a720581718c76428aa9b1006"}
Sep 08 07:29:55.060139 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:55.059967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99" event={"ID":"8d7362bc-2b5a-408e-9a09-b9ef28888d43","Type":"ContainerStarted","Data":"c4f3f85eb149c32d6f570bfcc635003600a05e83b0fc5e11cec57689f046a4a8"}
Sep 08 07:29:55.075062 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:55.075006    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99" podStartSLOduration=2.074985848 podStartE2EDuration="2.074985848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:29:55.073047498 +0000 UTC m=+1510.756507651" watchObservedRunningTime="2026-09-08 07:29:55.074985848 +0000 UTC m=+1510.758446002"
Sep 08 07:29:56.066540 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:56.066506    2580 generic.go:356] "Generic (PLEG): container finished" podID="8d7362bc-2b5a-408e-9a09-b9ef28888d43" containerID="bd5f7bb3932313cfe0afa0a139eb1ce62d527ba2a720581718c76428aa9b1006" exitCode=0
Sep 08 07:29:56.066966 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:56.066541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99" event={"ID":"8d7362bc-2b5a-408e-9a09-b9ef28888d43","Type":"ContainerDied","Data":"bd5f7bb3932313cfe0afa0a139eb1ce62d527ba2a720581718c76428aa9b1006"}
Sep 08 07:29:57.199715 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:57.199687    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"
Sep 08 07:29:57.296565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:57.296528    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7362bc-2b5a-408e-9a09-b9ef28888d43-maas-api-service-ca\") pod \"8d7362bc-2b5a-408e-9a09-b9ef28888d43\" (UID: \"8d7362bc-2b5a-408e-9a09-b9ef28888d43\") "
Sep 08 07:29:57.296899 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:57.296874    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7362bc-2b5a-408e-9a09-b9ef28888d43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d7362bc-2b5a-408e-9a09-b9ef28888d43" (UID: "8d7362bc-2b5a-408e-9a09-b9ef28888d43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:29:57.397552 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:57.397444    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7362bc-2b5a-408e-9a09-b9ef28888d43-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:29:58.077943 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:58.077912    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"
Sep 08 07:29:58.077943 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:58.077925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99" event={"ID":"8d7362bc-2b5a-408e-9a09-b9ef28888d43","Type":"ContainerDied","Data":"c4f3f85eb149c32d6f570bfcc635003600a05e83b0fc5e11cec57689f046a4a8"}
Sep 08 07:29:58.078187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:29:58.077974    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4f3f85eb149c32d6f570bfcc635003600a05e83b0fc5e11cec57689f046a4a8"
Sep 08 07:30:00.136274 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.136237    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"]
Sep 08 07:30:00.136711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.136683    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d7362bc-2b5a-408e-9a09-b9ef28888d43" containerName="revoke-keys"
Sep 08 07:30:00.136711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.136697    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7362bc-2b5a-408e-9a09-b9ef28888d43" containerName="revoke-keys"
Sep 08 07:30:00.136811 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.136766    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d7362bc-2b5a-408e-9a09-b9ef28888d43" containerName="revoke-keys"
Sep 08 07:30:00.139874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.139852    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"]
Sep 08 07:30:00.140010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.139973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"
Sep 08 07:30:00.142650 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.142631    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:30:00.143074 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.143056    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"
Sep 08 07:30:00.143617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.143596    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"]
Sep 08 07:30:00.147635 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.147615    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"]
Sep 08 07:30:00.147748 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.147708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"
Sep 08 07:30:00.152925 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.152904    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"]
Sep 08 07:30:00.156616 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.156597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"
Sep 08 07:30:00.157511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.157491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"]
Sep 08 07:30:00.191914 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.191887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"
Sep 08 07:30:00.201085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.201055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"
Sep 08 07:30:00.447397 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.447368    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"]
Sep 08 07:30:00.450098 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:30:00.450071    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f5be26_84c1_4aea_bdc5_c508b030387e.slice/crio-324e49377cbfb9b978a40b099ed63be87ebfd1309c628e1aad49d535cf14ce30 WatchSource:0}: Error finding container 324e49377cbfb9b978a40b099ed63be87ebfd1309c628e1aad49d535cf14ce30: Status 404 returned error can't find the container with id 324e49377cbfb9b978a40b099ed63be87ebfd1309c628e1aad49d535cf14ce30
Sep 08 07:30:00.484867 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.484842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"]
Sep 08 07:30:00.486337 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:30:00.486314    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a6c537_6806_4fef_b5e2_6d7965ced02c.slice/crio-51d94e1192eba0d1a3503aeaedd1f3a555bf62ec80f0ff36007c41744aff1dff WatchSource:0}: Error finding container 51d94e1192eba0d1a3503aeaedd1f3a555bf62ec80f0ff36007c41744aff1dff: Status 404 returned error can't find the container with id 51d94e1192eba0d1a3503aeaedd1f3a555bf62ec80f0ff36007c41744aff1dff
Sep 08 07:30:00.495502 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:00.495139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"]
Sep 08 07:30:01.098249 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.098209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr" event={"ID":"325af558-6f9e-47ae-af92-898a4cb480a3","Type":"ContainerStarted","Data":"a00bea219fc74f575f2017960b8bcf5e077742d2799e98ab2df5960be729f4d0"}
Sep 08 07:30:01.098440 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.098275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr" event={"ID":"325af558-6f9e-47ae-af92-898a4cb480a3","Type":"ContainerStarted","Data":"0e445ee07867237eecd98d2016b342469ca66c2cb47e7c6736f300c920e8a410"}
Sep 08 07:30:01.099878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.099850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx" event={"ID":"41a6c537-6806-4fef-b5e2-6d7965ced02c","Type":"ContainerStarted","Data":"fb6c5e393a91d3e2a899acff69172e4d2f639b631554fb3dc1a0e39cccb7edd1"}
Sep 08 07:30:01.099993 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.099901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx" event={"ID":"41a6c537-6806-4fef-b5e2-6d7965ced02c","Type":"ContainerStarted","Data":"51d94e1192eba0d1a3503aeaedd1f3a555bf62ec80f0ff36007c41744aff1dff"}
Sep 08 07:30:01.101448 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.101421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82" event={"ID":"06f5be26-84c1-4aea-bdc5-c508b030387e","Type":"ContainerStarted","Data":"d8435a4cde317a9802bab1f614a6055186454eeeb297c72d12d79b968041d95d"}
Sep 08 07:30:01.101548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.101484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82" event={"ID":"06f5be26-84c1-4aea-bdc5-c508b030387e","Type":"ContainerStarted","Data":"324e49377cbfb9b978a40b099ed63be87ebfd1309c628e1aad49d535cf14ce30"}
Sep 08 07:30:01.112644 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.112602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr" podStartSLOduration=1.112589052 podStartE2EDuration="1.112589052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:30:01.111702996 +0000 UTC m=+1516.795163151" watchObservedRunningTime="2026-09-08 07:30:01.112589052 +0000 UTC m=+1516.796049205"
Sep 08 07:30:01.125743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.125279    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx" podStartSLOduration=1.125261174 podStartE2EDuration="1.125261174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:30:01.12407174 +0000 UTC m=+1516.807531893" watchObservedRunningTime="2026-09-08 07:30:01.125261174 +0000 UTC m=+1516.808721328"
Sep 08 07:30:01.137697 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:01.137660    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82" podStartSLOduration=1.137646471 podStartE2EDuration="1.137646471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:30:01.1365925 +0000 UTC m=+1516.820052656" watchObservedRunningTime="2026-09-08 07:30:01.137646471 +0000 UTC m=+1516.821106663"
Sep 08 07:30:02.109030 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:02.108995    2580 generic.go:356] "Generic (PLEG): container finished" podID="325af558-6f9e-47ae-af92-898a4cb480a3" containerID="a00bea219fc74f575f2017960b8bcf5e077742d2799e98ab2df5960be729f4d0" exitCode=0
Sep 08 07:30:02.109255 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:02.109079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr" event={"ID":"325af558-6f9e-47ae-af92-898a4cb480a3","Type":"ContainerDied","Data":"a00bea219fc74f575f2017960b8bcf5e077742d2799e98ab2df5960be729f4d0"}
Sep 08 07:30:02.110655 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:02.110630    2580 generic.go:356] "Generic (PLEG): container finished" podID="41a6c537-6806-4fef-b5e2-6d7965ced02c" containerID="fb6c5e393a91d3e2a899acff69172e4d2f639b631554fb3dc1a0e39cccb7edd1" exitCode=0
Sep 08 07:30:02.110780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:02.110710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx" event={"ID":"41a6c537-6806-4fef-b5e2-6d7965ced02c","Type":"ContainerDied","Data":"fb6c5e393a91d3e2a899acff69172e4d2f639b631554fb3dc1a0e39cccb7edd1"}
Sep 08 07:30:02.112424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:02.112404    2580 generic.go:356] "Generic (PLEG): container finished" podID="06f5be26-84c1-4aea-bdc5-c508b030387e" containerID="d8435a4cde317a9802bab1f614a6055186454eeeb297c72d12d79b968041d95d" exitCode=0
Sep 08 07:30:02.112530 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:02.112438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82" event={"ID":"06f5be26-84c1-4aea-bdc5-c508b030387e","Type":"ContainerDied","Data":"d8435a4cde317a9802bab1f614a6055186454eeeb297c72d12d79b968041d95d"}
Sep 08 07:30:03.274216 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:03.274189    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"
Sep 08 07:30:03.297694 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:03.297662    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"
Sep 08 07:30:03.300792 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:03.300768    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"
Sep 08 07:30:04.127695 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.127654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82" event={"ID":"06f5be26-84c1-4aea-bdc5-c508b030387e","Type":"ContainerDied","Data":"324e49377cbfb9b978a40b099ed63be87ebfd1309c628e1aad49d535cf14ce30"}
Sep 08 07:30:04.127883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.127711    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="324e49377cbfb9b978a40b099ed63be87ebfd1309c628e1aad49d535cf14ce30"
Sep 08 07:30:04.127883 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.127665    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"
Sep 08 07:30:04.129101 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.129080    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"
Sep 08 07:30:04.129101 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.129092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr" event={"ID":"325af558-6f9e-47ae-af92-898a4cb480a3","Type":"ContainerDied","Data":"0e445ee07867237eecd98d2016b342469ca66c2cb47e7c6736f300c920e8a410"}
Sep 08 07:30:04.129286 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.129139    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e445ee07867237eecd98d2016b342469ca66c2cb47e7c6736f300c920e8a410"
Sep 08 07:30:04.130511 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.130480    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"
Sep 08 07:30:04.130665 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.130510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx" event={"ID":"41a6c537-6806-4fef-b5e2-6d7965ced02c","Type":"ContainerDied","Data":"51d94e1192eba0d1a3503aeaedd1f3a555bf62ec80f0ff36007c41744aff1dff"}
Sep 08 07:30:04.130665 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.130548    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51d94e1192eba0d1a3503aeaedd1f3a555bf62ec80f0ff36007c41744aff1dff"
Sep 08 07:30:04.638979 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.638944    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"]
Sep 08 07:30:04.639558 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639536    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f5be26-84c1-4aea-bdc5-c508b030387e" containerName="cleanup"
Sep 08 07:30:04.639558 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639560    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f5be26-84c1-4aea-bdc5-c508b030387e" containerName="cleanup"
Sep 08 07:30:04.639715 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639571    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a6c537-6806-4fef-b5e2-6d7965ced02c" containerName="cleanup"
Sep 08 07:30:04.639715 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639579    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a6c537-6806-4fef-b5e2-6d7965ced02c" containerName="cleanup"
Sep 08 07:30:04.639715 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639602    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325af558-6f9e-47ae-af92-898a4cb480a3" containerName="cleanup"
Sep 08 07:30:04.639715 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639609    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="325af558-6f9e-47ae-af92-898a4cb480a3" containerName="cleanup"
Sep 08 07:30:04.639715 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639715    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f5be26-84c1-4aea-bdc5-c508b030387e" containerName="cleanup"
Sep 08 07:30:04.639956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639726    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a6c537-6806-4fef-b5e2-6d7965ced02c" containerName="cleanup"
Sep 08 07:30:04.639956 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.639738    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="325af558-6f9e-47ae-af92-898a4cb480a3" containerName="cleanup"
Sep 08 07:30:04.642792 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.642772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.644657 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.644634    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fb59c2\"/\"rate-test-model-fb59c2-kserve-self-signed-certs\""
Sep 08 07:30:04.650536 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.650512    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"]
Sep 08 07:30:04.667831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.667791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:04.766543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.766500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-dshm\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.766746 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.766571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-home\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.766746 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.766626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tmp-dir\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.766746 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.766647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tls-certs\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.766746 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.766694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-model-cache\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.766897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.766772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjf2f\" (UniqueName: \"kubernetes.io/projected/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-kube-api-access-jjf2f\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.867231 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.867198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-dshm\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.867423 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.867251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-home\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.867423 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.867297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tmp-dir\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.867423 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.867313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tls-certs\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.867636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.867499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-model-cache\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.867636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.867553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjf2f\" (UniqueName: \"kubernetes.io/projected/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-kube-api-access-jjf2f\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.867736 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.867716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-home\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.868017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.867990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tmp-dir\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.868017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.868002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-model-cache\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.869770 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.869751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tls-certs\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.869854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.869837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-dshm\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.875356 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.875327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjf2f\" (UniqueName: \"kubernetes.io/projected/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-kube-api-access-jjf2f\") pod \"rate-test-model-fb59c2-kserve-79b5df897f-mrfl9\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") " pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:04.953156 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:04.953081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:05.227986 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:05.227960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"]
Sep 08 07:30:05.229966 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:30:05.229941    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode65aa7a8_63a5_4393_acc3_c7c43b153ce7.slice/crio-cabb822e8397e08ae59ff6a5b28178cd7320c11424a9181327d982c89de7e50c WatchSource:0}: Error finding container cabb822e8397e08ae59ff6a5b28178cd7320c11424a9181327d982c89de7e50c: Status 404 returned error can't find the container with id cabb822e8397e08ae59ff6a5b28178cd7320c11424a9181327d982c89de7e50c
Sep 08 07:30:06.143115 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:06.143082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9" event={"ID":"e65aa7a8-63a5-4393-acc3-c7c43b153ce7","Type":"ContainerStarted","Data":"9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0"}
Sep 08 07:30:06.143115 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:06.143122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9" event={"ID":"e65aa7a8-63a5-4393-acc3-c7c43b153ce7","Type":"ContainerStarted","Data":"cabb822e8397e08ae59ff6a5b28178cd7320c11424a9181327d982c89de7e50c"}
Sep 08 07:30:06.160653 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:06.160602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9" podStartSLOduration=2.160586542 podStartE2EDuration="2.160586542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:30:06.159580864 +0000 UTC m=+1521.843041018" watchObservedRunningTime="2026-09-08 07:30:06.160586542 +0000 UTC m=+1521.844046695"
Sep 08 07:30:06.748889 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:06.748847    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c" podUID="2f3eebf1-3f76-4470-8580-810716963701" containerName="main" containerID="cri-o://bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a" gracePeriod=45
Sep 08 07:30:07.000504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.000419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:30:07.089335 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz8mw\" (UniqueName: \"kubernetes.io/projected/2f3eebf1-3f76-4470-8580-810716963701-kube-api-access-fz8mw\") pod \"2f3eebf1-3f76-4470-8580-810716963701\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") "
Sep 08 07:30:07.089531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089359    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-home\") pod \"2f3eebf1-3f76-4470-8580-810716963701\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") "
Sep 08 07:30:07.089531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-model-cache\") pod \"2f3eebf1-3f76-4470-8580-810716963701\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") "
Sep 08 07:30:07.089531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3eebf1-3f76-4470-8580-810716963701-tls-certs\") pod \"2f3eebf1-3f76-4470-8580-810716963701\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") "
Sep 08 07:30:07.089531 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089506    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-dshm\") pod \"2f3eebf1-3f76-4470-8580-810716963701\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") "
Sep 08 07:30:07.089721 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089546    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-tmp-dir\") pod \"2f3eebf1-3f76-4470-8580-810716963701\" (UID: \"2f3eebf1-3f76-4470-8580-810716963701\") "
Sep 08 07:30:07.089721 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089690    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-home" (OuterVolumeSpecName: "home") pod "2f3eebf1-3f76-4470-8580-810716963701" (UID: "2f3eebf1-3f76-4470-8580-810716963701"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:30:07.089721 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f3eebf1-3f76-4470-8580-810716963701" (UID: "2f3eebf1-3f76-4470-8580-810716963701"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:30:07.089887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.089864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f3eebf1-3f76-4470-8580-810716963701" (UID: "2f3eebf1-3f76-4470-8580-810716963701"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:30:07.090089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.090058    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:07.090089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.090082    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:07.090260 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.090095    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:07.091793 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.091763    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-dshm" (OuterVolumeSpecName: "dshm") pod "2f3eebf1-3f76-4470-8580-810716963701" (UID: "2f3eebf1-3f76-4470-8580-810716963701"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:30:07.092068 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.092044    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3eebf1-3f76-4470-8580-810716963701-kube-api-access-fz8mw" (OuterVolumeSpecName: "kube-api-access-fz8mw") pod "2f3eebf1-3f76-4470-8580-810716963701" (UID: "2f3eebf1-3f76-4470-8580-810716963701"). InnerVolumeSpecName "kube-api-access-fz8mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:30:07.092143 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.092046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3eebf1-3f76-4470-8580-810716963701-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f3eebf1-3f76-4470-8580-810716963701" (UID: "2f3eebf1-3f76-4470-8580-810716963701"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:30:07.149578 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.149543    2580 generic.go:356] "Generic (PLEG): container finished" podID="2f3eebf1-3f76-4470-8580-810716963701" containerID="bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a" exitCode=0
Sep 08 07:30:07.150038 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.149592    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"
Sep 08 07:30:07.150038 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.149625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c" event={"ID":"2f3eebf1-3f76-4470-8580-810716963701","Type":"ContainerDied","Data":"bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a"}
Sep 08 07:30:07.150038 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.149666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c" event={"ID":"2f3eebf1-3f76-4470-8580-810716963701","Type":"ContainerDied","Data":"da95ab531c655d45d4429278b432151d5c5cb9e671a70ae49502a4f93b8a3b42"}
Sep 08 07:30:07.150038 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.149683    2580 scope.go:117] "RemoveContainer" containerID="bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a"
Sep 08 07:30:07.159014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.158993    2580 scope.go:117] "RemoveContainer" containerID="bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a"
Sep 08 07:30:07.159309 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:30:07.159288    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a\": container with ID starting with bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a not found: ID does not exist" containerID="bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a"
Sep 08 07:30:07.159376 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.159318    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a"} err="failed to get container status \"bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a\": rpc error: code = NotFound desc = could not find container \"bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a\": container with ID starting with bc46fa97213595e63713afa66afec41a470eec51c0078771bbecad3102bedc8a not found: ID does not exist"
Sep 08 07:30:07.170366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.170335    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"]
Sep 08 07:30:07.172129 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.172108    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/sub-test-model-fb59c2-kserve-6557965c49-95b5c"]
Sep 08 07:30:07.191452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.191416    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz8mw\" (UniqueName: \"kubernetes.io/projected/2f3eebf1-3f76-4470-8580-810716963701-kube-api-access-fz8mw\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:07.191452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.191445    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3eebf1-3f76-4470-8580-810716963701-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:07.191452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.191468    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3eebf1-3f76-4470-8580-810716963701-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:07.656365 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.656321    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz" podUID="11524b31-d7e2-4d9a-90b9-e57929f8248d" containerName="main" containerID="cri-o://18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1" gracePeriod=45
Sep 08 07:30:07.897847 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.897816    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:30:07.997900 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.997812    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-tmp-dir\") pod \"11524b31-d7e2-4d9a-90b9-e57929f8248d\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") "
Sep 08 07:30:07.997900 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.997862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11524b31-d7e2-4d9a-90b9-e57929f8248d-tls-certs\") pod \"11524b31-d7e2-4d9a-90b9-e57929f8248d\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") "
Sep 08 07:30:07.997900 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.997881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-dshm\") pod \"11524b31-d7e2-4d9a-90b9-e57929f8248d\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") "
Sep 08 07:30:07.997900 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.997900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-model-cache\") pod \"11524b31-d7e2-4d9a-90b9-e57929f8248d\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") "
Sep 08 07:30:07.998231 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.997928    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scbkp\" (UniqueName: \"kubernetes.io/projected/11524b31-d7e2-4d9a-90b9-e57929f8248d-kube-api-access-scbkp\") pod \"11524b31-d7e2-4d9a-90b9-e57929f8248d\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") "
Sep 08 07:30:07.998231 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.998001    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-home\") pod \"11524b31-d7e2-4d9a-90b9-e57929f8248d\" (UID: \"11524b31-d7e2-4d9a-90b9-e57929f8248d\") "
Sep 08 07:30:07.998231 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.998133    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11524b31-d7e2-4d9a-90b9-e57929f8248d" (UID: "11524b31-d7e2-4d9a-90b9-e57929f8248d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:30:07.998231 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.998147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-model-cache" (OuterVolumeSpecName: "model-cache") pod "11524b31-d7e2-4d9a-90b9-e57929f8248d" (UID: "11524b31-d7e2-4d9a-90b9-e57929f8248d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:30:07.998491 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.998446    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:07.998553 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.998486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-home" (OuterVolumeSpecName: "home") pod "11524b31-d7e2-4d9a-90b9-e57929f8248d" (UID: "11524b31-d7e2-4d9a-90b9-e57929f8248d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:30:07.998553 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:07.998495    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:08.000147 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.000120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11524b31-d7e2-4d9a-90b9-e57929f8248d-kube-api-access-scbkp" (OuterVolumeSpecName: "kube-api-access-scbkp") pod "11524b31-d7e2-4d9a-90b9-e57929f8248d" (UID: "11524b31-d7e2-4d9a-90b9-e57929f8248d"). InnerVolumeSpecName "kube-api-access-scbkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:30:08.000519 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.000500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11524b31-d7e2-4d9a-90b9-e57929f8248d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11524b31-d7e2-4d9a-90b9-e57929f8248d" (UID: "11524b31-d7e2-4d9a-90b9-e57929f8248d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:30:08.000601 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.000520    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-dshm" (OuterVolumeSpecName: "dshm") pod "11524b31-d7e2-4d9a-90b9-e57929f8248d" (UID: "11524b31-d7e2-4d9a-90b9-e57929f8248d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:30:08.099938 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.099893    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11524b31-d7e2-4d9a-90b9-e57929f8248d-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:08.099938 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.099930    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:08.099938 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.099939    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scbkp\" (UniqueName: \"kubernetes.io/projected/11524b31-d7e2-4d9a-90b9-e57929f8248d-kube-api-access-scbkp\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:08.099938 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.099948    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11524b31-d7e2-4d9a-90b9-e57929f8248d-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:08.157005 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.156972    2580 generic.go:356] "Generic (PLEG): container finished" podID="11524b31-d7e2-4d9a-90b9-e57929f8248d" containerID="18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1" exitCode=0
Sep 08 07:30:08.157400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.157015    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"
Sep 08 07:30:08.157400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.157059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz" event={"ID":"11524b31-d7e2-4d9a-90b9-e57929f8248d","Type":"ContainerDied","Data":"18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1"}
Sep 08 07:30:08.157400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.157098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz" event={"ID":"11524b31-d7e2-4d9a-90b9-e57929f8248d","Type":"ContainerDied","Data":"d510ef4e02e9014559f7081162b413ed912b0727510a9ece76447886591c3c94"}
Sep 08 07:30:08.157400 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.157116    2580 scope.go:117] "RemoveContainer" containerID="18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1"
Sep 08 07:30:08.172186 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.172156    2580 scope.go:117] "RemoveContainer" containerID="18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1"
Sep 08 07:30:08.172445 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:30:08.172426    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1\": container with ID starting with 18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1 not found: ID does not exist" containerID="18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1"
Sep 08 07:30:08.172507 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.172451    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1"} err="failed to get container status \"18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1\": rpc error: code = NotFound desc = could not find container \"18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1\": container with ID starting with 18b5c442974151faccd49a8197d1afec5b3b47f5740940dfca7098eaf55a3cd1 not found: ID does not exist"
Sep 08 07:30:08.181514 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.181486    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"]
Sep 08 07:30:08.187727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.187695    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/sub-test-model-217bc2-kserve-5b9974dd57-7hbrz"]
Sep 08 07:30:08.940804 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.940770    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11524b31-d7e2-4d9a-90b9-e57929f8248d" path="/var/lib/kubelet/pods/11524b31-d7e2-4d9a-90b9-e57929f8248d/volumes"
Sep 08 07:30:08.941222 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:08.941209    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f3eebf1-3f76-4470-8580-810716963701" path="/var/lib/kubelet/pods/2f3eebf1-3f76-4470-8580-810716963701/volumes"
Sep 08 07:30:09.485902 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:09.485871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:09.672831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:09.672791    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"]
Sep 08 07:30:09.678075 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:09.678050    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f0e0d8-zpk99"]
Sep 08 07:30:09.893643 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:09.893609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:09.990682 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:09.990649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:10.094156 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:10.094117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:10.940629 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:10.940596    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d7362bc-2b5a-408e-9a09-b9ef28888d43" path="/var/lib/kubelet/pods/8d7362bc-2b5a-408e-9a09-b9ef28888d43/volumes"
Sep 08 07:30:14.953206 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:14.953177    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:14.953703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:14.953378    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:14.965708 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:14.965687    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:15.205446 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:15.205369    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:30:16.372076 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.372041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:16.473416 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.473383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:16.625430 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.625349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:16.946992 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.946909    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"]
Sep 08 07:30:16.947264 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.947252    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3eebf1-3f76-4470-8580-810716963701" containerName="main"
Sep 08 07:30:16.947314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.947265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3eebf1-3f76-4470-8580-810716963701" containerName="main"
Sep 08 07:30:16.947314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.947284    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11524b31-d7e2-4d9a-90b9-e57929f8248d" containerName="main"
Sep 08 07:30:16.947314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.947289    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11524b31-d7e2-4d9a-90b9-e57929f8248d" containerName="main"
Sep 08 07:30:16.947411 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.947353    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3eebf1-3f76-4470-8580-810716963701" containerName="main"
Sep 08 07:30:16.947411 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.947363    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11524b31-d7e2-4d9a-90b9-e57929f8248d" containerName="main"
Sep 08 07:30:16.950190 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.950173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"
Sep 08 07:30:16.952144 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.952121    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:30:16.952144 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.952142    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:30:16.957250 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:16.957229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"]
Sep 08 07:30:17.091116 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:17.091073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59acfa10-04a5-49c1-8428-b016bbc6232f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk\" (UID: \"59acfa10-04a5-49c1-8428-b016bbc6232f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"
Sep 08 07:30:17.186932 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:17.186894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:17.192208 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:17.192168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59acfa10-04a5-49c1-8428-b016bbc6232f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk\" (UID: \"59acfa10-04a5-49c1-8428-b016bbc6232f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"
Sep 08 07:30:17.192847 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:17.192828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59acfa10-04a5-49c1-8428-b016bbc6232f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk\" (UID: \"59acfa10-04a5-49c1-8428-b016bbc6232f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"
Sep 08 07:30:17.261685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:17.261595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"
Sep 08 07:30:17.533863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:17.533837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"]
Sep 08 07:30:17.536521 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:30:17.536491    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59acfa10_04a5_49c1_8428_b016bbc6232f.slice/crio-e29acdc7f5bc81cab2cdf19eaf47af9894a98a36ea67a07143ee33928e0d8c87 WatchSource:0}: Error finding container e29acdc7f5bc81cab2cdf19eaf47af9894a98a36ea67a07143ee33928e0d8c87: Status 404 returned error can't find the container with id e29acdc7f5bc81cab2cdf19eaf47af9894a98a36ea67a07143ee33928e0d8c87
Sep 08 07:30:18.211529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:18.211482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk" event={"ID":"59acfa10-04a5-49c1-8428-b016bbc6232f","Type":"ContainerStarted","Data":"35d72fc6a355e5bfe805a03dec2cbefb7e9bf9688ea81b6f04c64f7addd1684e"}
Sep 08 07:30:18.211726 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:18.211550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk" event={"ID":"59acfa10-04a5-49c1-8428-b016bbc6232f","Type":"ContainerStarted","Data":"e29acdc7f5bc81cab2cdf19eaf47af9894a98a36ea67a07143ee33928e0d8c87"}
Sep 08 07:30:18.226688 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:18.226638    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk" podStartSLOduration=2.226621195 podStartE2EDuration="2.226621195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:30:18.22439836 +0000 UTC m=+1533.907858516" watchObservedRunningTime="2026-09-08 07:30:18.226621195 +0000 UTC m=+1533.910081393"
Sep 08 07:30:18.235869 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:30:18.235842    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59acfa10_04a5_49c1_8428_b016bbc6232f.slice/crio-conmon-35d72fc6a355e5bfe805a03dec2cbefb7e9bf9688ea81b6f04c64f7addd1684e.scope\": RecentStats: unable to find data in memory cache]"
Sep 08 07:30:19.218119 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:19.218084    2580 generic.go:356] "Generic (PLEG): container finished" podID="59acfa10-04a5-49c1-8428-b016bbc6232f" containerID="35d72fc6a355e5bfe805a03dec2cbefb7e9bf9688ea81b6f04c64f7addd1684e" exitCode=0
Sep 08 07:30:19.218528 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:19.218146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk" event={"ID":"59acfa10-04a5-49c1-8428-b016bbc6232f","Type":"ContainerDied","Data":"35d72fc6a355e5bfe805a03dec2cbefb7e9bf9688ea81b6f04c64f7addd1684e"}
Sep 08 07:30:20.358762 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:20.358732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"
Sep 08 07:30:20.417374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:20.417337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59acfa10-04a5-49c1-8428-b016bbc6232f-maas-api-service-ca\") pod \"59acfa10-04a5-49c1-8428-b016bbc6232f\" (UID: \"59acfa10-04a5-49c1-8428-b016bbc6232f\") "
Sep 08 07:30:20.417746 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:20.417721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59acfa10-04a5-49c1-8428-b016bbc6232f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59acfa10-04a5-49c1-8428-b016bbc6232f" (UID: "59acfa10-04a5-49c1-8428-b016bbc6232f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:30:20.518314 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:20.518221    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59acfa10-04a5-49c1-8428-b016bbc6232f-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:21.229639 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:21.229544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk" event={"ID":"59acfa10-04a5-49c1-8428-b016bbc6232f","Type":"ContainerDied","Data":"e29acdc7f5bc81cab2cdf19eaf47af9894a98a36ea67a07143ee33928e0d8c87"}
Sep 08 07:30:21.229639 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:21.229567    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"
Sep 08 07:30:21.229639 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:21.229596    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e29acdc7f5bc81cab2cdf19eaf47af9894a98a36ea67a07143ee33928e0d8c87"
Sep 08 07:30:24.297545 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:24.297513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:27.488453 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.488421    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"]
Sep 08 07:30:27.491152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.491109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:27.491290 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.491266    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-37abc6-29814210-c84xr"]
Sep 08 07:30:27.522725 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.522692    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"]
Sep 08 07:30:27.523006 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.522971    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd" podUID="6379ba01-5467-4d49-ab81-d3b828e9d639" containerName="payload-pre-processing" containerID="cri-o://fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c" gracePeriod=30
Sep 08 07:30:27.595741 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.595700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:27.697014 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.696982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:27.718377 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.718338    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"]
Sep 08 07:30:27.720318 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.720296    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-37abc6-9wfsk"]
Sep 08 07:30:27.761573 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.761545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:27.774135 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.774113    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:30:27.820040 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.820005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:27.886299 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.886262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlf42\" (UniqueName: \"kubernetes.io/projected/6379ba01-5467-4d49-ab81-d3b828e9d639-kube-api-access-hlf42\") pod \"6379ba01-5467-4d49-ab81-d3b828e9d639\" (UID: \"6379ba01-5467-4d49-ab81-d3b828e9d639\") "
Sep 08 07:30:27.886499 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.886326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6379ba01-5467-4d49-ab81-d3b828e9d639-plugins-config-volume\") pod \"6379ba01-5467-4d49-ab81-d3b828e9d639\" (UID: \"6379ba01-5467-4d49-ab81-d3b828e9d639\") "
Sep 08 07:30:27.886764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.886740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6379ba01-5467-4d49-ab81-d3b828e9d639-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6379ba01-5467-4d49-ab81-d3b828e9d639" (UID: "6379ba01-5467-4d49-ab81-d3b828e9d639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:30:27.888484 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.888443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6379ba01-5467-4d49-ab81-d3b828e9d639-kube-api-access-hlf42" (OuterVolumeSpecName: "kube-api-access-hlf42") pod "6379ba01-5467-4d49-ab81-d3b828e9d639" (UID: "6379ba01-5467-4d49-ab81-d3b828e9d639"). InnerVolumeSpecName "kube-api-access-hlf42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:30:27.987565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.987524    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6379ba01-5467-4d49-ab81-d3b828e9d639-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:27.987565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:27.987557    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlf42\" (UniqueName: \"kubernetes.io/projected/6379ba01-5467-4d49-ab81-d3b828e9d639-kube-api-access-hlf42\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:28.066570 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.066489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:28.274529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.274495    2580 generic.go:356] "Generic (PLEG): container finished" podID="6379ba01-5467-4d49-ab81-d3b828e9d639" containerID="fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c" exitCode=0
Sep 08 07:30:28.274701 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.274527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd" event={"ID":"6379ba01-5467-4d49-ab81-d3b828e9d639","Type":"ContainerDied","Data":"fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c"}
Sep 08 07:30:28.274701 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.274543    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"
Sep 08 07:30:28.274701 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.274583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd" event={"ID":"6379ba01-5467-4d49-ab81-d3b828e9d639","Type":"ContainerDied","Data":"6743acef3b25622b8691a8ff3f691a522a4af61ed4a1b2b2ff0b2c1bba711723"}
Sep 08 07:30:28.274701 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.274605    2580 scope.go:117] "RemoveContainer" containerID="fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c"
Sep 08 07:30:28.283251 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.283235    2580 scope.go:117] "RemoveContainer" containerID="fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c"
Sep 08 07:30:28.283529 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:30:28.283506    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c\": container with ID starting with fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c not found: ID does not exist" containerID="fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c"
Sep 08 07:30:28.283626 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.283535    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c"} err="failed to get container status \"fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c\": rpc error: code = NotFound desc = could not find container \"fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c\": container with ID starting with fd2ab9e18d5bbbbab45b06d682a59bc55c19b68024136aabbb9f68d391a5220c not found: ID does not exist"
Sep 08 07:30:28.294449 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.294423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"]
Sep 08 07:30:28.298287 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.298265    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-37abc6-5cb6895758-qkfpd"]
Sep 08 07:30:28.940018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.939983    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="325af558-6f9e-47ae-af92-898a4cb480a3" path="/var/lib/kubelet/pods/325af558-6f9e-47ae-af92-898a4cb480a3/volumes"
Sep 08 07:30:28.940404 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.940282    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59acfa10-04a5-49c1-8428-b016bbc6232f" path="/var/lib/kubelet/pods/59acfa10-04a5-49c1-8428-b016bbc6232f/volumes"
Sep 08 07:30:28.940608 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:28.940594    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6379ba01-5467-4d49-ab81-d3b828e9d639" path="/var/lib/kubelet/pods/6379ba01-5467-4d49-ab81-d3b828e9d639/volumes"
Sep 08 07:30:34.086644 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.086611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:34.360561 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.360473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:34.433637 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.433606    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:34.671437 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.671358    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"]
Sep 08 07:30:34.671725 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.671712    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59acfa10-04a5-49c1-8428-b016bbc6232f" containerName="revoke-keys"
Sep 08 07:30:34.671769 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.671726    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59acfa10-04a5-49c1-8428-b016bbc6232f" containerName="revoke-keys"
Sep 08 07:30:34.671769 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.671737    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6379ba01-5467-4d49-ab81-d3b828e9d639" containerName="payload-pre-processing"
Sep 08 07:30:34.671769 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.671742    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6379ba01-5467-4d49-ab81-d3b828e9d639" containerName="payload-pre-processing"
Sep 08 07:30:34.671870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.671816    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6379ba01-5467-4d49-ab81-d3b828e9d639" containerName="payload-pre-processing"
Sep 08 07:30:34.671870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.671828    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59acfa10-04a5-49c1-8428-b016bbc6232f" containerName="revoke-keys"
Sep 08 07:30:34.676299 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.676283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"
Sep 08 07:30:34.678566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.678544    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:30:34.678709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.678545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:30:34.683812 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.683624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"]
Sep 08 07:30:34.753483 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.753443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea92c51-83c0-480b-beda-3d129c0ac96a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r\" (UID: \"1ea92c51-83c0-480b-beda-3d129c0ac96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"
Sep 08 07:30:34.775582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.775548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:34.854629 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.854586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea92c51-83c0-480b-beda-3d129c0ac96a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r\" (UID: \"1ea92c51-83c0-480b-beda-3d129c0ac96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"
Sep 08 07:30:34.855298 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.855276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea92c51-83c0-480b-beda-3d129c0ac96a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r\" (UID: \"1ea92c51-83c0-480b-beda-3d129c0ac96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"
Sep 08 07:30:34.987450 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:34.987368    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"
Sep 08 07:30:35.275897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:35.275822    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"]
Sep 08 07:30:35.279218 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:30:35.279187    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea92c51_83c0_480b_beda_3d129c0ac96a.slice/crio-9d9d7c37ef217298fa0dc583ea9b2c0ef3d9fd93aeb9e19b8d8f6e69127ad522 WatchSource:0}: Error finding container 9d9d7c37ef217298fa0dc583ea9b2c0ef3d9fd93aeb9e19b8d8f6e69127ad522: Status 404 returned error can't find the container with id 9d9d7c37ef217298fa0dc583ea9b2c0ef3d9fd93aeb9e19b8d8f6e69127ad522
Sep 08 07:30:35.312742 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:35.312712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r" event={"ID":"1ea92c51-83c0-480b-beda-3d129c0ac96a","Type":"ContainerStarted","Data":"9d9d7c37ef217298fa0dc583ea9b2c0ef3d9fd93aeb9e19b8d8f6e69127ad522"}
Sep 08 07:30:36.319238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.319204    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ea92c51-83c0-480b-beda-3d129c0ac96a" containerID="3e1511922c3799ccf8cfd1867c590d6b5212d0d2fd0e9f43e8d515bc25a5ffeb" exitCode=0
Sep 08 07:30:36.319636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.319290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r" event={"ID":"1ea92c51-83c0-480b-beda-3d129c0ac96a","Type":"ContainerDied","Data":"3e1511922c3799ccf8cfd1867c590d6b5212d0d2fd0e9f43e8d515bc25a5ffeb"}
Sep 08 07:30:36.438996 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.438959    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"]
Sep 08 07:30:36.442873 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.442846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.445267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.445239    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"rate-test-model-217bc2-kserve-self-signed-certs\""
Sep 08 07:30:36.445504 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.445285    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"kube-root-ca.crt\""
Sep 08 07:30:36.445639 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.445285    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"default-dockercfg-r9gnf\""
Sep 08 07:30:36.445639 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.445297    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-217bc2\"/\"openshift-service-ca.crt\""
Sep 08 07:30:36.453054 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.453017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"]
Sep 08 07:30:36.471920 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.471890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:36.572488 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.572392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8s2\" (UniqueName: \"kubernetes.io/projected/d5640102-ff29-4db7-9495-b6234331e9d6-kube-api-access-xp8s2\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.572488 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.572427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-home\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.572668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.572536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-model-cache\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.572668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.572587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-tmp-dir\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.572668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.572633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5640102-ff29-4db7-9495-b6234331e9d6-tls-certs\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.572764 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.572671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-dshm\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674130 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-tmp-dir\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674130 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5640102-ff29-4db7-9495-b6234331e9d6-tls-certs\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-dshm\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8s2\" (UniqueName: \"kubernetes.io/projected/d5640102-ff29-4db7-9495-b6234331e9d6-kube-api-access-xp8s2\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-home\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-model-cache\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674611 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-tmp-dir\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-home\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.674703 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.674674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-model-cache\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.676961 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.676943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-dshm\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.677232 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.677214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5640102-ff29-4db7-9495-b6234331e9d6-tls-certs\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.682211 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.682189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8s2\" (UniqueName: \"kubernetes.io/projected/d5640102-ff29-4db7-9495-b6234331e9d6-kube-api-access-xp8s2\") pod \"rate-test-model-217bc2-kserve-956d48679-tc98s\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") " pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:36.755596 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:36.755555    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:37.027281 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:37.027251    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"]
Sep 08 07:30:37.029158 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:30:37.029129    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5640102_ff29_4db7_9495_b6234331e9d6.slice/crio-7e61560ba56fa8a2c3cdfcad3818169eef109e9eddce47d3072c52f7e1c43124 WatchSource:0}: Error finding container 7e61560ba56fa8a2c3cdfcad3818169eef109e9eddce47d3072c52f7e1c43124: Status 404 returned error can't find the container with id 7e61560ba56fa8a2c3cdfcad3818169eef109e9eddce47d3072c52f7e1c43124
Sep 08 07:30:37.326946 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:37.326910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s" event={"ID":"d5640102-ff29-4db7-9495-b6234331e9d6","Type":"ContainerStarted","Data":"72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa"}
Sep 08 07:30:37.327326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:37.326956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s" event={"ID":"d5640102-ff29-4db7-9495-b6234331e9d6","Type":"ContainerStarted","Data":"7e61560ba56fa8a2c3cdfcad3818169eef109e9eddce47d3072c52f7e1c43124"}
Sep 08 07:30:37.346585 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:37.346524    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s" podStartSLOduration=1.34650198 podStartE2EDuration="1.34650198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:30:37.345093513 +0000 UTC m=+1553.028553667" watchObservedRunningTime="2026-09-08 07:30:37.34650198 +0000 UTC m=+1553.029962134"
Sep 08 07:30:37.466675 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:37.466647    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"
Sep 08 07:30:37.583518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:37.583402    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea92c51-83c0-480b-beda-3d129c0ac96a-maas-api-service-ca\") pod \"1ea92c51-83c0-480b-beda-3d129c0ac96a\" (UID: \"1ea92c51-83c0-480b-beda-3d129c0ac96a\") "
Sep 08 07:30:37.583793 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:37.583770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea92c51-83c0-480b-beda-3d129c0ac96a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ea92c51-83c0-480b-beda-3d129c0ac96a" (UID: "1ea92c51-83c0-480b-beda-3d129c0ac96a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:30:37.684321 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:37.684280    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea92c51-83c0-480b-beda-3d129c0ac96a-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:38.333113 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:38.333081    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"
Sep 08 07:30:38.333525 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:38.333119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r" event={"ID":"1ea92c51-83c0-480b-beda-3d129c0ac96a","Type":"ContainerDied","Data":"9d9d7c37ef217298fa0dc583ea9b2c0ef3d9fd93aeb9e19b8d8f6e69127ad522"}
Sep 08 07:30:38.333525 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:38.333165    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d9d7c37ef217298fa0dc583ea9b2c0ef3d9fd93aeb9e19b8d8f6e69127ad522"
Sep 08 07:30:45.266737 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.266699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:45.275755 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.275726    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"]
Sep 08 07:30:45.281637 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.281608    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-0ca807-29814210-crm82"]
Sep 08 07:30:45.309444 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.309408    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"]
Sep 08 07:30:45.309672 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.309645    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc" podUID="41dbbc29-ac14-4cf6-86ef-03c317976256" containerName="payload-processing" containerID="cri-o://fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683" gracePeriod=30
Sep 08 07:30:45.321167 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.320835    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"]
Sep 08 07:30:45.321167 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.321066    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g" podUID="04b215da-5dde-4725-b707-18b8aa4737ad" containerName="payload-pre-processing" containerID="cri-o://0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c" gracePeriod=30
Sep 08 07:30:45.357562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.357527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:45.421834 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.421787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:45.521425 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.521370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:45.532676 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.532645    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"]
Sep 08 07:30:45.536636 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.536533    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ca807-4pt5r"]
Sep 08 07:30:45.604350 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.604319    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:30:45.605482 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.605427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:45.608268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.608243    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:30:45.759196 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.759164    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b215da-5dde-4725-b707-18b8aa4737ad-plugins-config-volume\") pod \"04b215da-5dde-4725-b707-18b8aa4737ad\" (UID: \"04b215da-5dde-4725-b707-18b8aa4737ad\") "
Sep 08 07:30:45.759478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.759209    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dbbc29-ac14-4cf6-86ef-03c317976256-plugins-config-volume\") pod \"41dbbc29-ac14-4cf6-86ef-03c317976256\" (UID: \"41dbbc29-ac14-4cf6-86ef-03c317976256\") "
Sep 08 07:30:45.759478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.759299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krrm5\" (UniqueName: \"kubernetes.io/projected/04b215da-5dde-4725-b707-18b8aa4737ad-kube-api-access-krrm5\") pod \"04b215da-5dde-4725-b707-18b8aa4737ad\" (UID: \"04b215da-5dde-4725-b707-18b8aa4737ad\") "
Sep 08 07:30:45.759478 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.759346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsvfc\" (UniqueName: \"kubernetes.io/projected/41dbbc29-ac14-4cf6-86ef-03c317976256-kube-api-access-bsvfc\") pod \"41dbbc29-ac14-4cf6-86ef-03c317976256\" (UID: \"41dbbc29-ac14-4cf6-86ef-03c317976256\") "
Sep 08 07:30:45.759624 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.759599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b215da-5dde-4725-b707-18b8aa4737ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04b215da-5dde-4725-b707-18b8aa4737ad" (UID: "04b215da-5dde-4725-b707-18b8aa4737ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:30:45.759741 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.759709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41dbbc29-ac14-4cf6-86ef-03c317976256-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41dbbc29-ac14-4cf6-86ef-03c317976256" (UID: "41dbbc29-ac14-4cf6-86ef-03c317976256"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:30:45.759808 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.759735    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b215da-5dde-4725-b707-18b8aa4737ad-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:45.761557 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.761520    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b215da-5dde-4725-b707-18b8aa4737ad-kube-api-access-krrm5" (OuterVolumeSpecName: "kube-api-access-krrm5") pod "04b215da-5dde-4725-b707-18b8aa4737ad" (UID: "04b215da-5dde-4725-b707-18b8aa4737ad"). InnerVolumeSpecName "kube-api-access-krrm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:30:45.761685 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.761577    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dbbc29-ac14-4cf6-86ef-03c317976256-kube-api-access-bsvfc" (OuterVolumeSpecName: "kube-api-access-bsvfc") pod "41dbbc29-ac14-4cf6-86ef-03c317976256" (UID: "41dbbc29-ac14-4cf6-86ef-03c317976256"). InnerVolumeSpecName "kube-api-access-bsvfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:30:45.860520 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.860412    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dbbc29-ac14-4cf6-86ef-03c317976256-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:45.860520 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.860451    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrm5\" (UniqueName: \"kubernetes.io/projected/04b215da-5dde-4725-b707-18b8aa4737ad-kube-api-access-krrm5\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:45.860520 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.860498    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsvfc\" (UniqueName: \"kubernetes.io/projected/41dbbc29-ac14-4cf6-86ef-03c317976256-kube-api-access-bsvfc\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:30:45.865484 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:45.865434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:46.378365 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.378331    2580 generic.go:356] "Generic (PLEG): container finished" podID="04b215da-5dde-4725-b707-18b8aa4737ad" containerID="0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c" exitCode=0
Sep 08 07:30:46.378843 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.378373    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"
Sep 08 07:30:46.378843 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.378419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g" event={"ID":"04b215da-5dde-4725-b707-18b8aa4737ad","Type":"ContainerDied","Data":"0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c"}
Sep 08 07:30:46.378843 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.378499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g" event={"ID":"04b215da-5dde-4725-b707-18b8aa4737ad","Type":"ContainerDied","Data":"8f6eb800e2b939dc1470bf8b17de61e111bd33dedb1591841387cbb6b3ffed4c"}
Sep 08 07:30:46.378843 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.378536    2580 scope.go:117] "RemoveContainer" containerID="0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c"
Sep 08 07:30:46.379964 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.379790    2580 generic.go:356] "Generic (PLEG): container finished" podID="41dbbc29-ac14-4cf6-86ef-03c317976256" containerID="fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683" exitCode=0
Sep 08 07:30:46.379964 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.379838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"
Sep 08 07:30:46.379964 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.379880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc" event={"ID":"41dbbc29-ac14-4cf6-86ef-03c317976256","Type":"ContainerDied","Data":"fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683"}
Sep 08 07:30:46.379964 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.379933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc" event={"ID":"41dbbc29-ac14-4cf6-86ef-03c317976256","Type":"ContainerDied","Data":"6ed93338e360478ddc459293cb8b9dfc1c7ad38a480774a22ca8e6c8f62cb575"}
Sep 08 07:30:46.389752 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.389646    2580 scope.go:117] "RemoveContainer" containerID="0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c"
Sep 08 07:30:46.390147 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:30:46.390127    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c\": container with ID starting with 0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c not found: ID does not exist" containerID="0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c"
Sep 08 07:30:46.390253 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.390153    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c"} err="failed to get container status \"0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c\": rpc error: code = NotFound desc = could not find container \"0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c\": container with ID starting with 0030e3681ac5d4509332644ba8b8e2f0eeb1dd2681c72c2b40cbde614e401b2c not found: ID does not exist"
Sep 08 07:30:46.390253 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.390171    2580 scope.go:117] "RemoveContainer" containerID="fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683"
Sep 08 07:30:46.398174 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.398156    2580 scope.go:117] "RemoveContainer" containerID="fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683"
Sep 08 07:30:46.398433 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:30:46.398417    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683\": container with ID starting with fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683 not found: ID does not exist" containerID="fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683"
Sep 08 07:30:46.398535 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.398439    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683"} err="failed to get container status \"fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683\": rpc error: code = NotFound desc = could not find container \"fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683\": container with ID starting with fcc67aec42e90c3fe41982a1d4a90bd0349667471b6e3a209931a929a44d6683 not found: ID does not exist"
Sep 08 07:30:46.410430 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.410407    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"]
Sep 08 07:30:46.417543 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.417522    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ca807-7b4d86c9c4-rl55g"]
Sep 08 07:30:46.428371 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.428343    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"]
Sep 08 07:30:46.433374 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.433355    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ca807-75bf5965bf-c9zdc"]
Sep 08 07:30:46.756343 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.756263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:46.756343 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.756310    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:46.768958 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.768935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:46.941098 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.941053    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b215da-5dde-4725-b707-18b8aa4737ad" path="/var/lib/kubelet/pods/04b215da-5dde-4725-b707-18b8aa4737ad/volumes"
Sep 08 07:30:46.941436 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.941416    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f5be26-84c1-4aea-bdc5-c508b030387e" path="/var/lib/kubelet/pods/06f5be26-84c1-4aea-bdc5-c508b030387e/volumes"
Sep 08 07:30:46.941794 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.941774    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ea92c51-83c0-480b-beda-3d129c0ac96a" path="/var/lib/kubelet/pods/1ea92c51-83c0-480b-beda-3d129c0ac96a/volumes"
Sep 08 07:30:46.942164 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:46.942146    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41dbbc29-ac14-4cf6-86ef-03c317976256" path="/var/lib/kubelet/pods/41dbbc29-ac14-4cf6-86ef-03c317976256/volumes"
Sep 08 07:30:47.399506 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:47.399476    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:30:52.775117 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:52.775081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:52.870734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:52.870703    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:30:53.071742 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:30:53.071709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:23.085847 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:23.085813    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"]
Sep 08 07:31:23.095854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:23.095822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:24.036487 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:24.036445    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"]
Sep 08 07:31:24.057529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:24.057498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:24.475534 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:24.475502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:24.773642 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:24.773567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:25.127451 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:25.127415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:28.021774 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:28.021738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:28.258908 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:28.258875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:29.186442 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.186410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:29.287056 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.287027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:29.709415 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.709384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:29.809800 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.809764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:29.823254 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823225    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"]
Sep 08 07:31:29.823582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823568    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ea92c51-83c0-480b-beda-3d129c0ac96a" containerName="revoke-keys"
Sep 08 07:31:29.823582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823581    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ea92c51-83c0-480b-beda-3d129c0ac96a" containerName="revoke-keys"
Sep 08 07:31:29.823672 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823604    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b215da-5dde-4725-b707-18b8aa4737ad" containerName="payload-pre-processing"
Sep 08 07:31:29.823672 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823610    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b215da-5dde-4725-b707-18b8aa4737ad" containerName="payload-pre-processing"
Sep 08 07:31:29.823672 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823617    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41dbbc29-ac14-4cf6-86ef-03c317976256" containerName="payload-processing"
Sep 08 07:31:29.823672 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823622    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41dbbc29-ac14-4cf6-86ef-03c317976256" containerName="payload-processing"
Sep 08 07:31:29.823797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823686    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41dbbc29-ac14-4cf6-86ef-03c317976256" containerName="payload-processing"
Sep 08 07:31:29.823797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823694    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ea92c51-83c0-480b-beda-3d129c0ac96a" containerName="revoke-keys"
Sep 08 07:31:29.823797 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.823703    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b215da-5dde-4725-b707-18b8aa4737ad" containerName="payload-pre-processing"
Sep 08 07:31:29.826837 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.826816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:29.828822 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.828798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4d980e\""
Sep 08 07:31:29.837017 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.836996    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"]
Sep 08 07:31:29.927557 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.927528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:29.973004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.972930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf\" (UID: \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:29.973004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:29.972972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wcd\" (UniqueName: \"kubernetes.io/projected/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-kube-api-access-65wcd\") pod \"payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf\" (UID: \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:30.074196 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.074161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf\" (UID: \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:30.074366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.074203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wcd\" (UniqueName: \"kubernetes.io/projected/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-kube-api-access-65wcd\") pod \"payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf\" (UID: \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:30.074900 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.074880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf\" (UID: \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:30.083600 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.083572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wcd\" (UniqueName: \"kubernetes.io/projected/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-kube-api-access-65wcd\") pod \"payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf\" (UID: \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:30.138418 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.138382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:30.415018 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.414990    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"]
Sep 08 07:31:30.417809 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:31:30.417776    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b14bf39_694e_4db6_8325_9e2f5e2bfe63.slice/crio-333627efb00aba3403db31922aa60b544716115e4dbcf94a8a639c198ea17656 WatchSource:0}: Error finding container 333627efb00aba3403db31922aa60b544716115e4dbcf94a8a639c198ea17656: Status 404 returned error can't find the container with id 333627efb00aba3403db31922aa60b544716115e4dbcf94a8a639c198ea17656
Sep 08 07:31:30.602108 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.602073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf" event={"ID":"5b14bf39-694e-4db6-8325-9e2f5e2bfe63","Type":"ContainerStarted","Data":"5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a"}
Sep 08 07:31:30.602309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.602124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf" event={"ID":"5b14bf39-694e-4db6-8325-9e2f5e2bfe63","Type":"ContainerStarted","Data":"333627efb00aba3403db31922aa60b544716115e4dbcf94a8a639c198ea17656"}
Sep 08 07:31:30.602309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.602165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:30.619596 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:30.619541    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf" podStartSLOduration=1.619523081 podStartE2EDuration="1.619523081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:31:30.61865727 +0000 UTC m=+1606.302117433" watchObservedRunningTime="2026-09-08 07:31:30.619523081 +0000 UTC m=+1606.302983246"
Sep 08 07:31:38.106626 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.106541    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9" podUID="e65aa7a8-63a5-4393-acc3-c7c43b153ce7" containerName="main" containerID="cri-o://9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0" gracePeriod=45
Sep 08 07:31:38.357428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.357367    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:31:38.453384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453344    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-home\") pod \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") "
Sep 08 07:31:38.453591 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-model-cache\") pod \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") "
Sep 08 07:31:38.453591 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453419    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjf2f\" (UniqueName: \"kubernetes.io/projected/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-kube-api-access-jjf2f\") pod \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") "
Sep 08 07:31:38.453591 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453482    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-dshm\") pod \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") "
Sep 08 07:31:38.453591 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453526    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tls-certs\") pod \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") "
Sep 08 07:31:38.453591 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tmp-dir\") pod \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\" (UID: \"e65aa7a8-63a5-4393-acc3-c7c43b153ce7\") "
Sep 08 07:31:38.453858 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453708    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-home" (OuterVolumeSpecName: "home") pod "e65aa7a8-63a5-4393-acc3-c7c43b153ce7" (UID: "e65aa7a8-63a5-4393-acc3-c7c43b153ce7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:31:38.453858 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453724    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e65aa7a8-63a5-4393-acc3-c7c43b153ce7" (UID: "e65aa7a8-63a5-4393-acc3-c7c43b153ce7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:31:38.453971 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453890    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:38.453971 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453904    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:38.453971 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.453937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e65aa7a8-63a5-4393-acc3-c7c43b153ce7" (UID: "e65aa7a8-63a5-4393-acc3-c7c43b153ce7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:31:38.455707 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.455681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-kube-api-access-jjf2f" (OuterVolumeSpecName: "kube-api-access-jjf2f") pod "e65aa7a8-63a5-4393-acc3-c7c43b153ce7" (UID: "e65aa7a8-63a5-4393-acc3-c7c43b153ce7"). InnerVolumeSpecName "kube-api-access-jjf2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:31:38.455707 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.455696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e65aa7a8-63a5-4393-acc3-c7c43b153ce7" (UID: "e65aa7a8-63a5-4393-acc3-c7c43b153ce7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:31:38.455880 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.455741    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-dshm" (OuterVolumeSpecName: "dshm") pod "e65aa7a8-63a5-4393-acc3-c7c43b153ce7" (UID: "e65aa7a8-63a5-4393-acc3-c7c43b153ce7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:31:38.555061 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.555014    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:38.555061 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.555052    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjf2f\" (UniqueName: \"kubernetes.io/projected/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-kube-api-access-jjf2f\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:38.555061 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.555066    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:38.555322 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.555080    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65aa7a8-63a5-4393-acc3-c7c43b153ce7-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:38.644694 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.644597    2580 generic.go:356] "Generic (PLEG): container finished" podID="e65aa7a8-63a5-4393-acc3-c7c43b153ce7" containerID="9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0" exitCode=0
Sep 08 07:31:38.644694 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.644636    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"
Sep 08 07:31:38.644694 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.644679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9" event={"ID":"e65aa7a8-63a5-4393-acc3-c7c43b153ce7","Type":"ContainerDied","Data":"9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0"}
Sep 08 07:31:38.644951 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.644724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9" event={"ID":"e65aa7a8-63a5-4393-acc3-c7c43b153ce7","Type":"ContainerDied","Data":"cabb822e8397e08ae59ff6a5b28178cd7320c11424a9181327d982c89de7e50c"}
Sep 08 07:31:38.644951 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.644742    2580 scope.go:117] "RemoveContainer" containerID="9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0"
Sep 08 07:31:38.653867 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.653849    2580 scope.go:117] "RemoveContainer" containerID="9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0"
Sep 08 07:31:38.654163 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:31:38.654141    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0\": container with ID starting with 9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0 not found: ID does not exist" containerID="9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0"
Sep 08 07:31:38.654235 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.654176    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0"} err="failed to get container status \"9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0\": rpc error: code = NotFound desc = could not find container \"9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0\": container with ID starting with 9d3d29dba043a78c6bab01643276017173b39972e94e466fd755118f88dd25e0 not found: ID does not exist"
Sep 08 07:31:38.666485 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.666440    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"]
Sep 08 07:31:38.668046 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.668024    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fb59c2/rate-test-model-fb59c2-kserve-79b5df897f-mrfl9"]
Sep 08 07:31:38.940072 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:38.939989    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e65aa7a8-63a5-4393-acc3-c7c43b153ce7" path="/var/lib/kubelet/pods/e65aa7a8-63a5-4393-acc3-c7c43b153ce7/volumes"
Sep 08 07:31:39.055780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.055717    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s" podUID="d5640102-ff29-4db7-9495-b6234331e9d6" containerName="main" containerID="cri-o://72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa" gracePeriod=45
Sep 08 07:31:39.301406 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.301384    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:31:39.363226 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-model-cache\") pod \"d5640102-ff29-4db7-9495-b6234331e9d6\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") "
Sep 08 07:31:39.363226 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363229    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-dshm\") pod \"d5640102-ff29-4db7-9495-b6234331e9d6\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") "
Sep 08 07:31:39.363428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp8s2\" (UniqueName: \"kubernetes.io/projected/d5640102-ff29-4db7-9495-b6234331e9d6-kube-api-access-xp8s2\") pod \"d5640102-ff29-4db7-9495-b6234331e9d6\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") "
Sep 08 07:31:39.363428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363290    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5640102-ff29-4db7-9495-b6234331e9d6-tls-certs\") pod \"d5640102-ff29-4db7-9495-b6234331e9d6\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") "
Sep 08 07:31:39.363428 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-home\") pod \"d5640102-ff29-4db7-9495-b6234331e9d6\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") "
Sep 08 07:31:39.363586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-tmp-dir\") pod \"d5640102-ff29-4db7-9495-b6234331e9d6\" (UID: \"d5640102-ff29-4db7-9495-b6234331e9d6\") "
Sep 08 07:31:39.363650 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5640102-ff29-4db7-9495-b6234331e9d6" (UID: "d5640102-ff29-4db7-9495-b6234331e9d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:31:39.363650 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363604    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-home" (OuterVolumeSpecName: "home") pod "d5640102-ff29-4db7-9495-b6234331e9d6" (UID: "d5640102-ff29-4db7-9495-b6234331e9d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:31:39.363781 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5640102-ff29-4db7-9495-b6234331e9d6" (UID: "d5640102-ff29-4db7-9495-b6234331e9d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:31:39.363891 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363877    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:39.363947 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363896    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:39.363947 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.363912    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:39.365453 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.365425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-dshm" (OuterVolumeSpecName: "dshm") pod "d5640102-ff29-4db7-9495-b6234331e9d6" (UID: "d5640102-ff29-4db7-9495-b6234331e9d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:31:39.365584 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.365503    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5640102-ff29-4db7-9495-b6234331e9d6-kube-api-access-xp8s2" (OuterVolumeSpecName: "kube-api-access-xp8s2") pod "d5640102-ff29-4db7-9495-b6234331e9d6" (UID: "d5640102-ff29-4db7-9495-b6234331e9d6"). InnerVolumeSpecName "kube-api-access-xp8s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:31:39.365584 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.365513    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5640102-ff29-4db7-9495-b6234331e9d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5640102-ff29-4db7-9495-b6234331e9d6" (UID: "d5640102-ff29-4db7-9495-b6234331e9d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:31:39.464887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.464849    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5640102-ff29-4db7-9495-b6234331e9d6-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:39.464887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.464880    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8s2\" (UniqueName: \"kubernetes.io/projected/d5640102-ff29-4db7-9495-b6234331e9d6-kube-api-access-xp8s2\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:39.464887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.464891    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5640102-ff29-4db7-9495-b6234331e9d6-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:31:39.650933 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.650901    2580 generic.go:356] "Generic (PLEG): container finished" podID="d5640102-ff29-4db7-9495-b6234331e9d6" containerID="72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa" exitCode=0
Sep 08 07:31:39.651152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.650943    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"
Sep 08 07:31:39.651152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.650995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s" event={"ID":"d5640102-ff29-4db7-9495-b6234331e9d6","Type":"ContainerDied","Data":"72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa"}
Sep 08 07:31:39.651152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.651043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s" event={"ID":"d5640102-ff29-4db7-9495-b6234331e9d6","Type":"ContainerDied","Data":"7e61560ba56fa8a2c3cdfcad3818169eef109e9eddce47d3072c52f7e1c43124"}
Sep 08 07:31:39.651152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.651063    2580 scope.go:117] "RemoveContainer" containerID="72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa"
Sep 08 07:31:39.660077 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.660046    2580 scope.go:117] "RemoveContainer" containerID="72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa"
Sep 08 07:31:39.660346 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:31:39.660327    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa\": container with ID starting with 72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa not found: ID does not exist" containerID="72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa"
Sep 08 07:31:39.660408 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.660353    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa"} err="failed to get container status \"72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa\": rpc error: code = NotFound desc = could not find container \"72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa\": container with ID starting with 72d5913cb6387298cd498f8eaaf57d93bbdf5976b6bf7e3836cf8846d64089aa not found: ID does not exist"
Sep 08 07:31:39.672668 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.672644    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"]
Sep 08 07:31:39.676819 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:39.676799    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-217bc2/rate-test-model-217bc2-kserve-956d48679-tc98s"]
Sep 08 07:31:40.939602 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:40.939569    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5640102-ff29-4db7-9495-b6234331e9d6" path="/var/lib/kubelet/pods/d5640102-ff29-4db7-9495-b6234331e9d6/volumes"
Sep 08 07:31:41.608878 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:41.608845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:31:52.716028 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:52.715993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:52.997181 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:52.997082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:53.679765 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:53.679728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:54.013946 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:54.013864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:54.271524 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:54.271422    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:56.902607 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:56.902576    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:56.994918 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:56.994878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:58.031984 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.031946    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"]
Sep 08 07:31:58.032515 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.032497    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65aa7a8-63a5-4393-acc3-c7c43b153ce7" containerName="main"
Sep 08 07:31:58.032589 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.032518    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65aa7a8-63a5-4393-acc3-c7c43b153ce7" containerName="main"
Sep 08 07:31:58.032589 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.032537    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5640102-ff29-4db7-9495-b6234331e9d6" containerName="main"
Sep 08 07:31:58.032589 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.032546    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5640102-ff29-4db7-9495-b6234331e9d6" containerName="main"
Sep 08 07:31:58.032753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.032649    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65aa7a8-63a5-4393-acc3-c7c43b153ce7" containerName="main"
Sep 08 07:31:58.032753 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.032664    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5640102-ff29-4db7-9495-b6234331e9d6" containerName="main"
Sep 08 07:31:58.035596 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.035576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.037452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.037436    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-79da86-serving-cert\""
Sep 08 07:31:58.037848 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.037827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qfrsm\""
Sep 08 07:31:58.038267 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.038249    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 07:31:58.038355 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.038271    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:31:58.038574 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.038552    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 07:31:58.046583 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.046391    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"]
Sep 08 07:31:58.079420 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.079393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:58.143278 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.143243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvdp8\" (UniqueName: \"kubernetes.io/projected/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-kube-api-access-gvdp8\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.143419 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.143307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-metrics-certs\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.143419 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.143393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.143544 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.143454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.244377 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.244342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.244581 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.244387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvdp8\" (UniqueName: \"kubernetes.io/projected/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-kube-api-access-gvdp8\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.244581 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.244431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-metrics-certs\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.244581 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.244500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.244581 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:31:58.244534    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-serving-cert: secret "maas-api-e2e-ipp-b-79da86-serving-cert" not found
Sep 08 07:31:58.244728 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:31:58.244625    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls podName:66b3d2bd-1a2f-4e20-9aa0-24081db41e28 nodeName:}" failed. No retries permitted until 2026-09-08 07:31:58.744604536 +0000 UTC m=+1634.428064685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls") pod "maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj" (UID: "66b3d2bd-1a2f-4e20-9aa0-24081db41e28") : secret "maas-api-e2e-ipp-b-79da86-serving-cert" not found
Sep 08 07:31:58.245133 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.245105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.246793 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.246774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-metrics-certs\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.252176 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.252155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvdp8\" (UniqueName: \"kubernetes.io/projected/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-kube-api-access-gvdp8\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.491026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.490997    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:58.689924 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.689892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:58.749319 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.749230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.751614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.751588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.790049 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.790011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:31:58.796611 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.796586    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"]
Sep 08 07:31:58.800296 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.800273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:58.805293 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.805269    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-79da86\""
Sep 08 07:31:58.805411 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.805307    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-79da86-dockercfg-ql6sv\""
Sep 08 07:31:58.815721 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.815698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"]
Sep 08 07:31:58.819490 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.819469    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"]
Sep 08 07:31:58.822761 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.822744    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:58.831836 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.831815    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"]
Sep 08 07:31:58.949237 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.949212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:31:58.951588 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.951560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp64s\" (UniqueName: \"kubernetes.io/projected/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-kube-api-access-sp64s\") pod \"payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v\" (UID: \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:58.951727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.951601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b84d98-2ea3-4543-a6a8-67c382e41780-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2\" (UID: \"b2b84d98-2ea3-4543-a6a8-67c382e41780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:58.951727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.951626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvpph\" (UniqueName: \"kubernetes.io/projected/b2b84d98-2ea3-4543-a6a8-67c382e41780-kube-api-access-rvpph\") pod \"payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2\" (UID: \"b2b84d98-2ea3-4543-a6a8-67c382e41780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:58.951727 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:58.951684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v\" (UID: \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:59.053246 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.053163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp64s\" (UniqueName: \"kubernetes.io/projected/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-kube-api-access-sp64s\") pod \"payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v\" (UID: \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:59.053246 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.053211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b84d98-2ea3-4543-a6a8-67c382e41780-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2\" (UID: \"b2b84d98-2ea3-4543-a6a8-67c382e41780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:59.053686 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.053489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvpph\" (UniqueName: \"kubernetes.io/projected/b2b84d98-2ea3-4543-a6a8-67c382e41780-kube-api-access-rvpph\") pod \"payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2\" (UID: \"b2b84d98-2ea3-4543-a6a8-67c382e41780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:59.053686 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.053582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v\" (UID: \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:59.053955 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.053938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b84d98-2ea3-4543-a6a8-67c382e41780-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2\" (UID: \"b2b84d98-2ea3-4543-a6a8-67c382e41780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:59.054209 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.054191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v\" (UID: \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:59.060935 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.060913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp64s\" (UniqueName: \"kubernetes.io/projected/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-kube-api-access-sp64s\") pod \"payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v\" (UID: \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:59.060935 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.060930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvpph\" (UniqueName: \"kubernetes.io/projected/b2b84d98-2ea3-4543-a6a8-67c382e41780-kube-api-access-rvpph\") pod \"payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2\" (UID: \"b2b84d98-2ea3-4543-a6a8-67c382e41780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:59.110355 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.110317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:59.132780 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.132749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:59.221048 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.221022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"]
Sep 08 07:31:59.223407 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:31:59.223381    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b3d2bd_1a2f_4e20_9aa0_24081db41e28.slice/crio-b8b0a52a42d10500e1e33440b14450c33eea59a064d95bd175454bcb989b03b7 WatchSource:0}: Error finding container b8b0a52a42d10500e1e33440b14450c33eea59a064d95bd175454bcb989b03b7: Status 404 returned error can't find the container with id b8b0a52a42d10500e1e33440b14450c33eea59a064d95bd175454bcb989b03b7
Sep 08 07:31:59.225257 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.225236    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 07:31:59.383292 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.383257    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"]
Sep 08 07:31:59.385685 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:31:59.385650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c643b8_ca4e_4186_9483_c8eb5d4a56b8.slice/crio-0e7eefa87ee186cf0fca61ef7b95dd64b3f525346de026c71251c15f917a1bb7 WatchSource:0}: Error finding container 0e7eefa87ee186cf0fca61ef7b95dd64b3f525346de026c71251c15f917a1bb7: Status 404 returned error can't find the container with id 0e7eefa87ee186cf0fca61ef7b95dd64b3f525346de026c71251c15f917a1bb7
Sep 08 07:31:59.411509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.411485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"]
Sep 08 07:31:59.413221 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:31:59.413190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b84d98_2ea3_4543_a6a8_67c382e41780.slice/crio-7c56fc328b2a4fb7573b9f1dfb261c0ab131ebd79d66ad1bd6d78e33ea418bad WatchSource:0}: Error finding container 7c56fc328b2a4fb7573b9f1dfb261c0ab131ebd79d66ad1bd6d78e33ea418bad: Status 404 returned error can't find the container with id 7c56fc328b2a4fb7573b9f1dfb261c0ab131ebd79d66ad1bd6d78e33ea418bad
Sep 08 07:31:59.758916 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.758702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v" event={"ID":"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8","Type":"ContainerStarted","Data":"5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6"}
Sep 08 07:31:59.758916 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.758772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v" event={"ID":"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8","Type":"ContainerStarted","Data":"0e7eefa87ee186cf0fca61ef7b95dd64b3f525346de026c71251c15f917a1bb7"}
Sep 08 07:31:59.758916 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.758801    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:31:59.760611 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.760583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2" event={"ID":"b2b84d98-2ea3-4543-a6a8-67c382e41780","Type":"ContainerStarted","Data":"530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704"}
Sep 08 07:31:59.760744 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.760632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2" event={"ID":"b2b84d98-2ea3-4543-a6a8-67c382e41780","Type":"ContainerStarted","Data":"7c56fc328b2a4fb7573b9f1dfb261c0ab131ebd79d66ad1bd6d78e33ea418bad"}
Sep 08 07:31:59.760744 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.760675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:31:59.762430 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.762403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj" event={"ID":"66b3d2bd-1a2f-4e20-9aa0-24081db41e28","Type":"ContainerStarted","Data":"b8b0a52a42d10500e1e33440b14450c33eea59a064d95bd175454bcb989b03b7"}
Sep 08 07:31:59.775962 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.775901    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v" podStartSLOduration=1.775885119 podStartE2EDuration="1.775885119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:31:59.773847739 +0000 UTC m=+1635.457307915" watchObservedRunningTime="2026-09-08 07:31:59.775885119 +0000 UTC m=+1635.459345271"
Sep 08 07:31:59.791615 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:31:59.791563    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2" podStartSLOduration=1.791543187 podStartE2EDuration="1.791543187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:31:59.789569923 +0000 UTC m=+1635.473030078" watchObservedRunningTime="2026-09-08 07:31:59.791543187 +0000 UTC m=+1635.475003642"
Sep 08 07:32:00.769157 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:00.769124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj" event={"ID":"66b3d2bd-1a2f-4e20-9aa0-24081db41e28","Type":"ContainerStarted","Data":"ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a"}
Sep 08 07:32:00.769665 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:00.769371    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:32:00.787750 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:00.787705    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj" podStartSLOduration=1.343438424 podStartE2EDuration="2.787689705s" totalImagesPullingTime="1.444251281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:32:00.786811139 +0000 UTC m=+1636.470271291" watchObservedRunningTime="2026-09-08 07:32:00.787689705 +0000 UTC m=+1636.471149865"
Sep 08 07:32:06.781870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:06.781838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:32:10.771317 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:10.771291    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:32:10.771714 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:10.771392    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:32:13.948441 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:13.948413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:32:21.873085 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:21.873052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:32:26.962169 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:26.962141    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2_b2b84d98-2ea3-4543-a6a8-67c382e41780/payload-processing/0.log"
Sep 08 07:32:37.339255 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.339226    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"]
Sep 08 07:32:37.342684 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.342664    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.344853 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.344812    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4d980e\"/\"kube-root-ca.crt\""
Sep 08 07:32:37.345723 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.345704    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4d980e\"/\"ipp-route-4d980e-kserve-self-signed-certs\""
Sep 08 07:32:37.345840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.345743    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4d980e\"/\"default-dockercfg-6rjwb\""
Sep 08 07:32:37.345840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.345762    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4d980e\"/\"openshift-service-ca.crt\""
Sep 08 07:32:37.354323 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.354301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"]
Sep 08 07:32:37.397216 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.397181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-model-cache\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.397379 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.397245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-dshm\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.397379 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.397322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-home\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.397379 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.397366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57566f62-1a86-4cac-873d-6db6d49111f7-tls-certs\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.397529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.397406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-tmp-dir\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.397529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.397433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkx7h\" (UniqueName: \"kubernetes.io/projected/57566f62-1a86-4cac-873d-6db6d49111f7-kube-api-access-xkx7h\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.415859 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.415833    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:32:37.498341 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-dshm\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.498533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-home\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.498533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57566f62-1a86-4cac-873d-6db6d49111f7-tls-certs\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.498533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-tmp-dir\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.498533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkx7h\" (UniqueName: \"kubernetes.io/projected/57566f62-1a86-4cac-873d-6db6d49111f7-kube-api-access-xkx7h\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.498533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-model-cache\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.498869 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-tmp-dir\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.498869 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-home\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.499021 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.498900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-model-cache\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.500650 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.500630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-dshm\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.500955 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.500939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57566f62-1a86-4cac-873d-6db6d49111f7-tls-certs\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.507055 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.507030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkx7h\" (UniqueName: \"kubernetes.io/projected/57566f62-1a86-4cac-873d-6db6d49111f7-kube-api-access-xkx7h\") pod \"ipp-route-4d980e-kserve-585c84968c-hkkqc\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") " pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.651897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.651809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:37.930176 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.930098    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"]
Sep 08 07:32:37.935475 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:32:37.935432    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57566f62_1a86_4cac_873d_6db6d49111f7.slice/crio-aaddb1b36fc73835b126e84946a5105d6be49e4bf592c52731ae3d492f7a2688 WatchSource:0}: Error finding container aaddb1b36fc73835b126e84946a5105d6be49e4bf592c52731ae3d492f7a2688: Status 404 returned error can't find the container with id aaddb1b36fc73835b126e84946a5105d6be49e4bf592c52731ae3d492f7a2688
Sep 08 07:32:37.963890 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:37.963863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc" event={"ID":"57566f62-1a86-4cac-873d-6db6d49111f7","Type":"ContainerStarted","Data":"aaddb1b36fc73835b126e84946a5105d6be49e4bf592c52731ae3d492f7a2688"}
Sep 08 07:32:38.970362 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:38.970324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc" event={"ID":"57566f62-1a86-4cac-873d-6db6d49111f7","Type":"ContainerStarted","Data":"e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4"}
Sep 08 07:32:38.990393 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:38.990340    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc" podStartSLOduration=1.990323521 podStartE2EDuration="1.990323521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:32:38.989274374 +0000 UTC m=+1674.672734528" watchObservedRunningTime="2026-09-08 07:32:38.990323521 +0000 UTC m=+1674.673783673"
Sep 08 07:32:45.119268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:45.119239    2580 scope.go:117] "RemoveContainer" containerID="7213653edf8028d010062701d2ec0d608da7677afcc9d1d90268b874c525e645"
Sep 08 07:32:47.652169 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:47.652108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:47.652626 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:47.652193    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:47.664779 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:47.664751    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:32:48.027495 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:32:48.027382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:33:02.759853 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:02.759822    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2_b2b84d98-2ea3-4543-a6a8-67c382e41780/payload-processing/0.log"
Sep 08 07:33:03.047931 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:03.047844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:03.363652 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:03.363616    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:03.605729 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:03.605695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:06.533116 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:06.533083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:06.629575 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:06.629539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:07.652943 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.652855    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"]
Sep 08 07:33:07.656395 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.656337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.659211 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.659189    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-dd2695-serving-cert\""
Sep 08 07:33:07.668023 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.667998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"]
Sep 08 07:33:07.677026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.677002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.677209 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.677066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds44n\" (UniqueName: \"kubernetes.io/projected/5fc8827e-f083-4669-a8ca-d7840cb9abb0-kube-api-access-ds44n\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.677660 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.677234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8827e-f083-4669-a8ca-d7840cb9abb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.677660 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.677292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.691578 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.691552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:07.778582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.778551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8827e-f083-4669-a8ca-d7840cb9abb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.778841 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.778591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.778841 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.778635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.778841 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.778665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds44n\" (UniqueName: \"kubernetes.io/projected/5fc8827e-f083-4669-a8ca-d7840cb9abb0-kube-api-access-ds44n\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.778841 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:33:07.778786    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-serving-cert: secret "maas-api-e2e-ipp-cleanup-dd2695-serving-cert" not found
Sep 08 07:33:07.779100 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:33:07.778875    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls podName:5fc8827e-f083-4669-a8ca-d7840cb9abb0 nodeName:}" failed. No retries permitted until 2026-09-08 07:33:08.278854864 +0000 UTC m=+1703.962314997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr" (UID: "5fc8827e-f083-4669-a8ca-d7840cb9abb0") : secret "maas-api-e2e-ipp-cleanup-dd2695-serving-cert" not found
Sep 08 07:33:07.779211 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.779192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8827e-f083-4669-a8ca-d7840cb9abb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.781004 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.780985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.787383 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.787357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds44n\" (UniqueName: \"kubernetes.io/projected/5fc8827e-f083-4669-a8ca-d7840cb9abb0-kube-api-access-ds44n\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:07.795089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:07.795062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:08.080652 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:08.080622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:08.271538 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:08.271508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:08.283763 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:08.283735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:08.286564 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:08.286540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:08.370433 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:08.370354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:08.570404 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:08.570369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:08.841438 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:08.841290    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"]
Sep 08 07:33:08.844018 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:33:08.843982    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc8827e_f083_4669_a8ca_d7840cb9abb0.slice/crio-4705bbfd758f85572ea5b7b6d26919fde167471199cf23f69c4e7f949dbc5d1b WatchSource:0}: Error finding container 4705bbfd758f85572ea5b7b6d26919fde167471199cf23f69c4e7f949dbc5d1b: Status 404 returned error can't find the container with id 4705bbfd758f85572ea5b7b6d26919fde167471199cf23f69c4e7f949dbc5d1b
Sep 08 07:33:09.125960 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:09.125920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr" event={"ID":"5fc8827e-f083-4669-a8ca-d7840cb9abb0","Type":"ContainerStarted","Data":"4705bbfd758f85572ea5b7b6d26919fde167471199cf23f69c4e7f949dbc5d1b"}
Sep 08 07:33:10.132811 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:10.132781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr" event={"ID":"5fc8827e-f083-4669-a8ca-d7840cb9abb0","Type":"ContainerStarted","Data":"3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927"}
Sep 08 07:33:10.133214 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:10.132842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:10.151498 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:10.151430    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr" podStartSLOduration=2.786224032 podStartE2EDuration="3.151411954s" totalImagesPullingTime="365.187922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:33:10.149171811 +0000 UTC m=+1705.832631961" watchObservedRunningTime="2026-09-08 07:33:10.151411954 +0000 UTC m=+1705.834872106"
Sep 08 07:33:16.143659 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:16.143628    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:31.459286 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:31.459251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:38.191109 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.191068    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"]
Sep 08 07:33:38.191756 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.191290    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr" podUID="5fc8827e-f083-4669-a8ca-d7840cb9abb0" containerName="maas-api" containerID="cri-o://3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927" gracePeriod=30
Sep 08 07:33:38.227568 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.227529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:38.304671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.304637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:38.361566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.361478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:38.439876 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.439849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:38.443424 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.443370    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:38.570563 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.570533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-metrics-certs\") pod \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") "
Sep 08 07:33:38.570743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.570583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8827e-f083-4669-a8ca-d7840cb9abb0-openshift-service-ca\") pod \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") "
Sep 08 07:33:38.570743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.570665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls\") pod \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") "
Sep 08 07:33:38.570743 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.570685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds44n\" (UniqueName: \"kubernetes.io/projected/5fc8827e-f083-4669-a8ca-d7840cb9abb0-kube-api-access-ds44n\") pod \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\" (UID: \"5fc8827e-f083-4669-a8ca-d7840cb9abb0\") "
Sep 08 07:33:38.571010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.570981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc8827e-f083-4669-a8ca-d7840cb9abb0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fc8827e-f083-4669-a8ca-d7840cb9abb0" (UID: "5fc8827e-f083-4669-a8ca-d7840cb9abb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:33:38.572833 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.572807    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5fc8827e-f083-4669-a8ca-d7840cb9abb0" (UID: "5fc8827e-f083-4669-a8ca-d7840cb9abb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:33:38.572948 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.572921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc8827e-f083-4669-a8ca-d7840cb9abb0-kube-api-access-ds44n" (OuterVolumeSpecName: "kube-api-access-ds44n") pod "5fc8827e-f083-4669-a8ca-d7840cb9abb0" (UID: "5fc8827e-f083-4669-a8ca-d7840cb9abb0"). InnerVolumeSpecName "kube-api-access-ds44n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:33:38.572948 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.572921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fc8827e-f083-4669-a8ca-d7840cb9abb0" (UID: "5fc8827e-f083-4669-a8ca-d7840cb9abb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:33:38.672338 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.672299    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8827e-f083-4669-a8ca-d7840cb9abb0-openshift-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:33:38.672338 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.672330    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-maas-api-tls\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:33:38.672338 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.672341    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds44n\" (UniqueName: \"kubernetes.io/projected/5fc8827e-f083-4669-a8ca-d7840cb9abb0-kube-api-access-ds44n\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:33:38.672338 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.672349    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fc8827e-f083-4669-a8ca-d7840cb9abb0-metrics-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:33:38.800648 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.800563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:38.859494 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:38.859448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:39.280600 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.280564    2580 generic.go:356] "Generic (PLEG): container finished" podID="5fc8827e-f083-4669-a8ca-d7840cb9abb0" containerID="3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927" exitCode=0
Sep 08 07:33:39.281052 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.280622    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"
Sep 08 07:33:39.281052 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.280634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr" event={"ID":"5fc8827e-f083-4669-a8ca-d7840cb9abb0","Type":"ContainerDied","Data":"3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927"}
Sep 08 07:33:39.281052 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.280679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr" event={"ID":"5fc8827e-f083-4669-a8ca-d7840cb9abb0","Type":"ContainerDied","Data":"4705bbfd758f85572ea5b7b6d26919fde167471199cf23f69c4e7f949dbc5d1b"}
Sep 08 07:33:39.281052 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.280707    2580 scope.go:117] "RemoveContainer" containerID="3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927"
Sep 08 07:33:39.289612 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.289590    2580 scope.go:117] "RemoveContainer" containerID="3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927"
Sep 08 07:33:39.289875 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:33:39.289855    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927\": container with ID starting with 3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927 not found: ID does not exist" containerID="3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927"
Sep 08 07:33:39.289915 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.289886    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927"} err="failed to get container status \"3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927\": rpc error: code = NotFound desc = could not find container \"3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927\": container with ID starting with 3b52c53b21fafa345f8c72ae714a7be5bc52811477281bf45305e15e02f4d927 not found: ID does not exist"
Sep 08 07:33:39.297554 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.297530    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"]
Sep 08 07:33:39.304857 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:39.304833    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dd2695-579d55b54d-h9ssr"]
Sep 08 07:33:40.941260 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:40.941225    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fc8827e-f083-4669-a8ca-d7840cb9abb0" path="/var/lib/kubelet/pods/5fc8827e-f083-4669-a8ca-d7840cb9abb0/volumes"
Sep 08 07:33:45.322405 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:45.322376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:45.448421 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:45.448341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:45.581577 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:45.581546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:33:49.181840 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:33:49.181794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:04.592602 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:04.592564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:04.628344 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:04.628306    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"]
Sep 08 07:34:04.628617 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:04.628581    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf" podUID="5b14bf39-694e-4db6-8325-9e2f5e2bfe63" containerName="payload-pre-processing" containerID="cri-o://5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a" gracePeriod=30
Sep 08 07:34:04.658064 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:04.658036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:04.719826 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:04.719788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:04.807477 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:04.807431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:04.867839 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:04.867808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:04.885558 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:04.885538    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:34:05.023340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.023299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-plugins-config-volume\") pod \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\" (UID: \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\") "
Sep 08 07:34:05.023548 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.023351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65wcd\" (UniqueName: \"kubernetes.io/projected/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-kube-api-access-65wcd\") pod \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\" (UID: \"5b14bf39-694e-4db6-8325-9e2f5e2bfe63\") "
Sep 08 07:34:05.023783 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.023756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b14bf39-694e-4db6-8325-9e2f5e2bfe63" (UID: "5b14bf39-694e-4db6-8325-9e2f5e2bfe63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:34:05.025603 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.025572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-kube-api-access-65wcd" (OuterVolumeSpecName: "kube-api-access-65wcd") pod "5b14bf39-694e-4db6-8325-9e2f5e2bfe63" (UID: "5b14bf39-694e-4db6-8325-9e2f5e2bfe63"). InnerVolumeSpecName "kube-api-access-65wcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:34:05.125109 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.125026    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:34:05.125109 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.125054    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65wcd\" (UniqueName: \"kubernetes.io/projected/5b14bf39-694e-4db6-8325-9e2f5e2bfe63-kube-api-access-65wcd\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:34:05.189887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.189848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:05.419481 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.419365    2580 generic.go:356] "Generic (PLEG): container finished" podID="5b14bf39-694e-4db6-8325-9e2f5e2bfe63" containerID="5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a" exitCode=0
Sep 08 07:34:05.419481 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.419416    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"
Sep 08 07:34:05.419481 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.419447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf" event={"ID":"5b14bf39-694e-4db6-8325-9e2f5e2bfe63","Type":"ContainerDied","Data":"5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a"}
Sep 08 07:34:05.419729 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.419515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf" event={"ID":"5b14bf39-694e-4db6-8325-9e2f5e2bfe63","Type":"ContainerDied","Data":"333627efb00aba3403db31922aa60b544716115e4dbcf94a8a639c198ea17656"}
Sep 08 07:34:05.419729 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.419537    2580 scope.go:117] "RemoveContainer" containerID="5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a"
Sep 08 07:34:05.428499 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.428476    2580 scope.go:117] "RemoveContainer" containerID="5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a"
Sep 08 07:34:05.428793 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:34:05.428775    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a\": container with ID starting with 5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a not found: ID does not exist" containerID="5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a"
Sep 08 07:34:05.428850 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.428801    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a"} err="failed to get container status \"5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a\": rpc error: code = NotFound desc = could not find container \"5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a\": container with ID starting with 5dc0e713aaac84d3b1e348aab873d78e74715ede205f42248c8588fc9403953a not found: ID does not exist"
Sep 08 07:34:05.440562 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.440516    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"]
Sep 08 07:34:05.442366 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:05.442341    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4d980e-6dc65fdf7d-fldzf"]
Sep 08 07:34:06.939984 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:06.939953    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b14bf39-694e-4db6-8325-9e2f5e2bfe63" path="/var/lib/kubelet/pods/5b14bf39-694e-4db6-8325-9e2f5e2bfe63/volumes"
Sep 08 07:34:10.178245 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:10.178209    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"]
Sep 08 07:34:10.273129 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:10.273093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:23.038402 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:23.038357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:25.197751 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.197690    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc" podUID="57566f62-1a86-4cac-873d-6db6d49111f7" containerName="main" containerID="cri-o://e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4" gracePeriod=45
Sep 08 07:34:25.447152 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.447125    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:34:25.519863 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.519769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57566f62-1a86-4cac-873d-6db6d49111f7-tls-certs\") pod \"57566f62-1a86-4cac-873d-6db6d49111f7\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") "
Sep 08 07:34:25.520040 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.519890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-model-cache\") pod \"57566f62-1a86-4cac-873d-6db6d49111f7\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") "
Sep 08 07:34:25.520040 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.519938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-tmp-dir\") pod \"57566f62-1a86-4cac-873d-6db6d49111f7\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") "
Sep 08 07:34:25.520040 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.519974    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-home\") pod \"57566f62-1a86-4cac-873d-6db6d49111f7\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") "
Sep 08 07:34:25.520040 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.520001    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkx7h\" (UniqueName: \"kubernetes.io/projected/57566f62-1a86-4cac-873d-6db6d49111f7-kube-api-access-xkx7h\") pod \"57566f62-1a86-4cac-873d-6db6d49111f7\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") "
Sep 08 07:34:25.520040 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.520037    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-dshm\") pod \"57566f62-1a86-4cac-873d-6db6d49111f7\" (UID: \"57566f62-1a86-4cac-873d-6db6d49111f7\") "
Sep 08 07:34:25.520325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.520160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "57566f62-1a86-4cac-873d-6db6d49111f7" (UID: "57566f62-1a86-4cac-873d-6db6d49111f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:34:25.520325 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.520230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57566f62-1a86-4cac-873d-6db6d49111f7" (UID: "57566f62-1a86-4cac-873d-6db6d49111f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:34:25.520443 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.520348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-home" (OuterVolumeSpecName: "home") pod "57566f62-1a86-4cac-873d-6db6d49111f7" (UID: "57566f62-1a86-4cac-873d-6db6d49111f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:34:25.520536 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.520490    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:34:25.520536 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.520509    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:34:25.520645 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.520536    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:34:25.522556 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.522506    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57566f62-1a86-4cac-873d-6db6d49111f7-kube-api-access-xkx7h" (OuterVolumeSpecName: "kube-api-access-xkx7h") pod "57566f62-1a86-4cac-873d-6db6d49111f7" (UID: "57566f62-1a86-4cac-873d-6db6d49111f7"). InnerVolumeSpecName "kube-api-access-xkx7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:34:25.522691 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.522580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57566f62-1a86-4cac-873d-6db6d49111f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57566f62-1a86-4cac-873d-6db6d49111f7" (UID: "57566f62-1a86-4cac-873d-6db6d49111f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:34:25.522758 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.522740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-dshm" (OuterVolumeSpecName: "dshm") pod "57566f62-1a86-4cac-873d-6db6d49111f7" (UID: "57566f62-1a86-4cac-873d-6db6d49111f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:34:25.523008 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.522984    2580 generic.go:356] "Generic (PLEG): container finished" podID="57566f62-1a86-4cac-873d-6db6d49111f7" containerID="e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4" exitCode=0
Sep 08 07:34:25.523073 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.523022    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"
Sep 08 07:34:25.523073 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.523042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc" event={"ID":"57566f62-1a86-4cac-873d-6db6d49111f7","Type":"ContainerDied","Data":"e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4"}
Sep 08 07:34:25.523184 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.523080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc" event={"ID":"57566f62-1a86-4cac-873d-6db6d49111f7","Type":"ContainerDied","Data":"aaddb1b36fc73835b126e84946a5105d6be49e4bf592c52731ae3d492f7a2688"}
Sep 08 07:34:25.523184 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.523103    2580 scope.go:117] "RemoveContainer" containerID="e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4"
Sep 08 07:34:25.541982 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.541959    2580 scope.go:117] "RemoveContainer" containerID="e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4"
Sep 08 07:34:25.542335 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:34:25.542304    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4\": container with ID starting with e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4 not found: ID does not exist" containerID="e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4"
Sep 08 07:34:25.542426 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.542346    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4"} err="failed to get container status \"e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4\": rpc error: code = NotFound desc = could not find container \"e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4\": container with ID starting with e8fbc822ef994b98f3aeeb4da17a29b6f580fd4e074593b9e0ea53933fb6c7d4 not found: ID does not exist"
Sep 08 07:34:25.551572 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.551550    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"]
Sep 08 07:34:25.554921 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.554900    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4d980e/ipp-route-4d980e-kserve-585c84968c-hkkqc"]
Sep 08 07:34:25.620976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.620927    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkx7h\" (UniqueName: \"kubernetes.io/projected/57566f62-1a86-4cac-873d-6db6d49111f7-kube-api-access-xkx7h\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:34:25.620976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.620971    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57566f62-1a86-4cac-873d-6db6d49111f7-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:34:25.620976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:25.620982    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57566f62-1a86-4cac-873d-6db6d49111f7-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:34:26.939663 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:26.939632    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57566f62-1a86-4cac-873d-6db6d49111f7" path="/var/lib/kubelet/pods/57566f62-1a86-4cac-873d-6db6d49111f7/volumes"
Sep 08 07:34:29.173136 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:29.173094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:35.074521 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:35.074483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:35.174934 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:35.174900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:45.149822 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:45.149790    2580 scope.go:117] "RemoveContainer" containerID="63bdae98e3090c682a587bd19b6ff86ced8f5dda91846a40778a895743da4c75"
Sep 08 07:34:46.650089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:46.650047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:47.799500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:47.799448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:51.374450 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:51.374412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:53.722661 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:53.722622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:53.788665 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:53.788618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:54.032620 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:54.032532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:34:54.479952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:34:54.479912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:04.921304 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:04.921269    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"]
Sep 08 07:35:04.923824 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:04.921513    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj" podUID="66b3d2bd-1a2f-4e20-9aa0-24081db41e28" containerName="maas-api" containerID="cri-o://ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a" gracePeriod=30
Sep 08 07:35:04.950888 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:04.950857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:04.978850 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:04.978816    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"]
Sep 08 07:35:04.979070 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:04.979040    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2" podUID="b2b84d98-2ea3-4543-a6a8-67c382e41780" containerName="payload-processing" containerID="cri-o://530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704" gracePeriod=30
Sep 08 07:35:04.987297 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:04.987210    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"]
Sep 08 07:35:04.987611 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:04.987574    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v" podUID="b1c643b8-ca4e-4186-9483-c8eb5d4a56b8" containerName="payload-pre-processing" containerID="cri-o://5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6" gracePeriod=30
Sep 08 07:35:05.258518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.258493    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:35:05.268176 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.268149    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:35:05.279172 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.279141    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:35:05.286371 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286343    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls\") pod \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") "
Sep 08 07:35:05.286371 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp64s\" (UniqueName: \"kubernetes.io/projected/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-kube-api-access-sp64s\") pod \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\" (UID: \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\") "
Sep 08 07:35:05.286700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b84d98-2ea3-4543-a6a8-67c382e41780-plugins-config-volume\") pod \"b2b84d98-2ea3-4543-a6a8-67c382e41780\" (UID: \"b2b84d98-2ea3-4543-a6a8-67c382e41780\") "
Sep 08 07:35:05.286700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-plugins-config-volume\") pod \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\" (UID: \"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8\") "
Sep 08 07:35:05.286700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-openshift-service-ca\") pod \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") "
Sep 08 07:35:05.286700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-metrics-certs\") pod \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") "
Sep 08 07:35:05.286700 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvpph\" (UniqueName: \"kubernetes.io/projected/b2b84d98-2ea3-4543-a6a8-67c382e41780-kube-api-access-rvpph\") pod \"b2b84d98-2ea3-4543-a6a8-67c382e41780\" (UID: \"b2b84d98-2ea3-4543-a6a8-67c382e41780\") "
Sep 08 07:35:05.286988 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvdp8\" (UniqueName: \"kubernetes.io/projected/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-kube-api-access-gvdp8\") pod \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\" (UID: \"66b3d2bd-1a2f-4e20-9aa0-24081db41e28\") "
Sep 08 07:35:05.286988 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.286784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b84d98-2ea3-4543-a6a8-67c382e41780-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2b84d98-2ea3-4543-a6a8-67c382e41780" (UID: "b2b84d98-2ea3-4543-a6a8-67c382e41780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:35:05.287516 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.287485    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66b3d2bd-1a2f-4e20-9aa0-24081db41e28" (UID: "66b3d2bd-1a2f-4e20-9aa0-24081db41e28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:35:05.288170 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.288127    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1c643b8-ca4e-4186-9483-c8eb5d4a56b8" (UID: "b1c643b8-ca4e-4186-9483-c8eb5d4a56b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:35:05.289734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.288860    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b84d98-2ea3-4543-a6a8-67c382e41780-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:05.289734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.288886    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:05.289734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.288907    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-openshift-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:05.289734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.289537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-kube-api-access-gvdp8" (OuterVolumeSpecName: "kube-api-access-gvdp8") pod "66b3d2bd-1a2f-4e20-9aa0-24081db41e28" (UID: "66b3d2bd-1a2f-4e20-9aa0-24081db41e28"). InnerVolumeSpecName "kube-api-access-gvdp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:35:05.289734 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.289613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-kube-api-access-sp64s" (OuterVolumeSpecName: "kube-api-access-sp64s") pod "b1c643b8-ca4e-4186-9483-c8eb5d4a56b8" (UID: "b1c643b8-ca4e-4186-9483-c8eb5d4a56b8"). InnerVolumeSpecName "kube-api-access-sp64s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:35:05.299447 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.299414    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66b3d2bd-1a2f-4e20-9aa0-24081db41e28" (UID: "66b3d2bd-1a2f-4e20-9aa0-24081db41e28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:35:05.300561 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.300526    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b84d98-2ea3-4543-a6a8-67c382e41780-kube-api-access-rvpph" (OuterVolumeSpecName: "kube-api-access-rvpph") pod "b2b84d98-2ea3-4543-a6a8-67c382e41780" (UID: "b2b84d98-2ea3-4543-a6a8-67c382e41780"). InnerVolumeSpecName "kube-api-access-rvpph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:35:05.301742 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.301719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66b3d2bd-1a2f-4e20-9aa0-24081db41e28" (UID: "66b3d2bd-1a2f-4e20-9aa0-24081db41e28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:35:05.390125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.390082    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-metrics-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:05.390125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.390118    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvpph\" (UniqueName: \"kubernetes.io/projected/b2b84d98-2ea3-4543-a6a8-67c382e41780-kube-api-access-rvpph\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:05.390125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.390131    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvdp8\" (UniqueName: \"kubernetes.io/projected/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-kube-api-access-gvdp8\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:05.390370 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.390144    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b3d2bd-1a2f-4e20-9aa0-24081db41e28-maas-api-tls\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:05.390370 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.390156    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp64s\" (UniqueName: \"kubernetes.io/projected/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8-kube-api-access-sp64s\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:05.723829 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.723792    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1c643b8-ca4e-4186-9483-c8eb5d4a56b8" containerID="5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6" exitCode=0
Sep 08 07:35:05.724034 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.723846    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"
Sep 08 07:35:05.724034 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.723883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v" event={"ID":"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8","Type":"ContainerDied","Data":"5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6"}
Sep 08 07:35:05.724034 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.723969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v" event={"ID":"b1c643b8-ca4e-4186-9483-c8eb5d4a56b8","Type":"ContainerDied","Data":"0e7eefa87ee186cf0fca61ef7b95dd64b3f525346de026c71251c15f917a1bb7"}
Sep 08 07:35:05.724034 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.723997    2580 scope.go:117] "RemoveContainer" containerID="5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6"
Sep 08 07:35:05.725367 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.725338    2580 generic.go:356] "Generic (PLEG): container finished" podID="b2b84d98-2ea3-4543-a6a8-67c382e41780" containerID="530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704" exitCode=0
Sep 08 07:35:05.725509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.725399    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"
Sep 08 07:35:05.725509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.725418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2" event={"ID":"b2b84d98-2ea3-4543-a6a8-67c382e41780","Type":"ContainerDied","Data":"530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704"}
Sep 08 07:35:05.725509 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.725483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2" event={"ID":"b2b84d98-2ea3-4543-a6a8-67c382e41780","Type":"ContainerDied","Data":"7c56fc328b2a4fb7573b9f1dfb261c0ab131ebd79d66ad1bd6d78e33ea418bad"}
Sep 08 07:35:05.727307 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.727285    2580 generic.go:356] "Generic (PLEG): container finished" podID="66b3d2bd-1a2f-4e20-9aa0-24081db41e28" containerID="ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a" exitCode=0
Sep 08 07:35:05.727521 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.727343    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"
Sep 08 07:35:05.727650 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.727361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj" event={"ID":"66b3d2bd-1a2f-4e20-9aa0-24081db41e28","Type":"ContainerDied","Data":"ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a"}
Sep 08 07:35:05.727725 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.727656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj" event={"ID":"66b3d2bd-1a2f-4e20-9aa0-24081db41e28","Type":"ContainerDied","Data":"b8b0a52a42d10500e1e33440b14450c33eea59a064d95bd175454bcb989b03b7"}
Sep 08 07:35:05.734020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.733937    2580 scope.go:117] "RemoveContainer" containerID="5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6"
Sep 08 07:35:05.734421 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:35:05.734394    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6\": container with ID starting with 5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6 not found: ID does not exist" containerID="5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6"
Sep 08 07:35:05.734559 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.734426    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6"} err="failed to get container status \"5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6\": rpc error: code = NotFound desc = could not find container \"5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6\": container with ID starting with 5dccf03bcc25fa833d511475a489078d5d06aa7bc55b7a0b34ae9ef25cd8a9c6 not found: ID does not exist"
Sep 08 07:35:05.734559 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.734442    2580 scope.go:117] "RemoveContainer" containerID="530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704"
Sep 08 07:35:05.744118 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.744095    2580 scope.go:117] "RemoveContainer" containerID="530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704"
Sep 08 07:35:05.744445 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:35:05.744425    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704\": container with ID starting with 530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704 not found: ID does not exist" containerID="530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704"
Sep 08 07:35:05.744555 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.744453    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704"} err="failed to get container status \"530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704\": rpc error: code = NotFound desc = could not find container \"530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704\": container with ID starting with 530cad473ddc3590ae526c851e1b9c5eca86083e7aadc228a6903ff54fc2e704 not found: ID does not exist"
Sep 08 07:35:05.744598 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.744559    2580 scope.go:117] "RemoveContainer" containerID="ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a"
Sep 08 07:35:05.748064 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.748042    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"]
Sep 08 07:35:05.753560 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.753537    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79da86-6579d899c4-bnj6v"]
Sep 08 07:35:05.753858 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.753844    2580 scope.go:117] "RemoveContainer" containerID="ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a"
Sep 08 07:35:05.754141 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:35:05.754126    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a\": container with ID starting with ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a not found: ID does not exist" containerID="ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a"
Sep 08 07:35:05.754187 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.754146    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a"} err="failed to get container status \"ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a\": rpc error: code = NotFound desc = could not find container \"ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a\": container with ID starting with ed759391c898d03916bc9f219e67016b73047ac0dbaf364e8eddb386f7d4190a not found: ID does not exist"
Sep 08 07:35:05.761785 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.761753    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"]
Sep 08 07:35:05.768193 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.768172    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-79da86-77788c5c95-kglsj"]
Sep 08 07:35:05.777614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.777593    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"]
Sep 08 07:35:05.781935 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:05.781914    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-79da86-6dffd599b5-rwpt2"]
Sep 08 07:35:06.942308 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:06.942268    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66b3d2bd-1a2f-4e20-9aa0-24081db41e28" path="/var/lib/kubelet/pods/66b3d2bd-1a2f-4e20-9aa0-24081db41e28/volumes"
Sep 08 07:35:06.942842 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:06.942824    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c643b8-ca4e-4186-9483-c8eb5d4a56b8" path="/var/lib/kubelet/pods/b1c643b8-ca4e-4186-9483-c8eb5d4a56b8/volumes"
Sep 08 07:35:06.943287 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:06.943271    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b84d98-2ea3-4543-a6a8-67c382e41780" path="/var/lib/kubelet/pods/b2b84d98-2ea3-4543-a6a8-67c382e41780/volumes"
Sep 08 07:35:07.608529 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:07.608494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:12.111622 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.111583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:12.186376 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.186335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:12.400071 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.399988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:12.740089 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740008    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"]
Sep 08 07:35:12.740349 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740337    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b14bf39-694e-4db6-8325-9e2f5e2bfe63" containerName="payload-pre-processing"
Sep 08 07:35:12.740390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740351    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b14bf39-694e-4db6-8325-9e2f5e2bfe63" containerName="payload-pre-processing"
Sep 08 07:35:12.740390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740364    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b84d98-2ea3-4543-a6a8-67c382e41780" containerName="payload-processing"
Sep 08 07:35:12.740390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740370    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b84d98-2ea3-4543-a6a8-67c382e41780" containerName="payload-processing"
Sep 08 07:35:12.740390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740376    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b3d2bd-1a2f-4e20-9aa0-24081db41e28" containerName="maas-api"
Sep 08 07:35:12.740390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740381    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b3d2bd-1a2f-4e20-9aa0-24081db41e28" containerName="maas-api"
Sep 08 07:35:12.740390 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740390    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc8827e-f083-4669-a8ca-d7840cb9abb0" containerName="maas-api"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740396    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc8827e-f083-4669-a8ca-d7840cb9abb0" containerName="maas-api"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740404    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c643b8-ca4e-4186-9483-c8eb5d4a56b8" containerName="payload-pre-processing"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740409    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c643b8-ca4e-4186-9483-c8eb5d4a56b8" containerName="payload-pre-processing"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740418    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57566f62-1a86-4cac-873d-6db6d49111f7" containerName="main"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740423    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="57566f62-1a86-4cac-873d-6db6d49111f7" containerName="main"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740493    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b14bf39-694e-4db6-8325-9e2f5e2bfe63" containerName="payload-pre-processing"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740505    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b84d98-2ea3-4543-a6a8-67c382e41780" containerName="payload-processing"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740518    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c643b8-ca4e-4186-9483-c8eb5d4a56b8" containerName="payload-pre-processing"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740525    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b3d2bd-1a2f-4e20-9aa0-24081db41e28" containerName="maas-api"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740533    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="57566f62-1a86-4cac-873d-6db6d49111f7" containerName="main"
Sep 08 07:35:12.740610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.740539    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fc8827e-f083-4669-a8ca-d7840cb9abb0" containerName="maas-api"
Sep 08 07:35:12.744971 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.744953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"
Sep 08 07:35:12.748214 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.748197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:35:12.748307 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.748218    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:35:12.760831 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.760807    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"]
Sep 08 07:35:12.861730 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.861695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d094ea7-9a2d-45b0-ab2c-cf49c525374a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg\" (UID: \"2d094ea7-9a2d-45b0-ab2c-cf49c525374a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"
Sep 08 07:35:12.894761 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.894734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:12.962315 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.962290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d094ea7-9a2d-45b0-ab2c-cf49c525374a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg\" (UID: \"2d094ea7-9a2d-45b0-ab2c-cf49c525374a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"
Sep 08 07:35:12.962870 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:12.962850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d094ea7-9a2d-45b0-ab2c-cf49c525374a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg\" (UID: \"2d094ea7-9a2d-45b0-ab2c-cf49c525374a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"
Sep 08 07:35:13.054333 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:13.054257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"
Sep 08 07:35:13.323874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:13.323851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"]
Sep 08 07:35:13.326554 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:35:13.326528    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d094ea7_9a2d_45b0_ab2c_cf49c525374a.slice/crio-d465d3752e0667ed29ba345f9ec900abc3d57ce8b2fed13772b0c4ef478072b1 WatchSource:0}: Error finding container d465d3752e0667ed29ba345f9ec900abc3d57ce8b2fed13772b0c4ef478072b1: Status 404 returned error can't find the container with id d465d3752e0667ed29ba345f9ec900abc3d57ce8b2fed13772b0c4ef478072b1
Sep 08 07:35:13.772188 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:13.772156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg" event={"ID":"2d094ea7-9a2d-45b0-ab2c-cf49c525374a","Type":"ContainerStarted","Data":"9e9d244e2cc4e62915b37b8f364cb9ecdf14107affeb183f68df2f0adcc3e113"}
Sep 08 07:35:13.772378 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:13.772206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg" event={"ID":"2d094ea7-9a2d-45b0-ab2c-cf49c525374a","Type":"ContainerStarted","Data":"d465d3752e0667ed29ba345f9ec900abc3d57ce8b2fed13772b0c4ef478072b1"}
Sep 08 07:35:13.789248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:13.789203    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg" podStartSLOduration=1.7891889029999999 podStartE2EDuration="1.789188903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:35:13.788345158 +0000 UTC m=+1829.471805312" watchObservedRunningTime="2026-09-08 07:35:13.789188903 +0000 UTC m=+1829.472649056"
Sep 08 07:35:14.778782 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:14.778750    2580 generic.go:356] "Generic (PLEG): container finished" podID="2d094ea7-9a2d-45b0-ab2c-cf49c525374a" containerID="9e9d244e2cc4e62915b37b8f364cb9ecdf14107affeb183f68df2f0adcc3e113" exitCode=0
Sep 08 07:35:14.779243 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:14.778833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg" event={"ID":"2d094ea7-9a2d-45b0-ab2c-cf49c525374a","Type":"ContainerDied","Data":"9e9d244e2cc4e62915b37b8f364cb9ecdf14107affeb183f68df2f0adcc3e113"}
Sep 08 07:35:15.909061 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:15.909035    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"
Sep 08 07:35:15.987902 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:15.987872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d094ea7-9a2d-45b0-ab2c-cf49c525374a-maas-api-service-ca\") pod \"2d094ea7-9a2d-45b0-ab2c-cf49c525374a\" (UID: \"2d094ea7-9a2d-45b0-ab2c-cf49c525374a\") "
Sep 08 07:35:15.988215 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:15.988194    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d094ea7-9a2d-45b0-ab2c-cf49c525374a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d094ea7-9a2d-45b0-ab2c-cf49c525374a" (UID: "2d094ea7-9a2d-45b0-ab2c-cf49c525374a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:35:16.089071 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:16.089033    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d094ea7-9a2d-45b0-ab2c-cf49c525374a-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:16.791825 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:16.789725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg" event={"ID":"2d094ea7-9a2d-45b0-ab2c-cf49c525374a","Type":"ContainerDied","Data":"d465d3752e0667ed29ba345f9ec900abc3d57ce8b2fed13772b0c4ef478072b1"}
Sep 08 07:35:16.791825 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:16.789788    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d465d3752e0667ed29ba345f9ec900abc3d57ce8b2fed13772b0c4ef478072b1"
Sep 08 07:35:16.791825 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:16.789851    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"
Sep 08 07:35:23.324521 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.324454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:23.398310 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.398278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"]
Sep 08 07:35:23.398573 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.398541    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h" podUID="cdf6b61d-77c8-4610-8022-531bbe7ffd63" containerName="payload-pre-processing" containerID="cri-o://5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1" gracePeriod=30
Sep 08 07:35:23.429364 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.429339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:23.525794 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.525766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:23.617570 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.617497    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"]
Sep 08 07:35:23.619002 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.618976    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fb59c2-v8qhg"]
Sep 08 07:35:23.646492 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.646442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:23.647384 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.647367    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:35:23.707112 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.707082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:23.759294 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.759257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sb8k\" (UniqueName: \"kubernetes.io/projected/cdf6b61d-77c8-4610-8022-531bbe7ffd63-kube-api-access-8sb8k\") pod \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\" (UID: \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\") "
Sep 08 07:35:23.759500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.759349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6b61d-77c8-4610-8022-531bbe7ffd63-plugins-config-volume\") pod \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\" (UID: \"cdf6b61d-77c8-4610-8022-531bbe7ffd63\") "
Sep 08 07:35:23.759766 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.759740    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf6b61d-77c8-4610-8022-531bbe7ffd63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdf6b61d-77c8-4610-8022-531bbe7ffd63" (UID: "cdf6b61d-77c8-4610-8022-531bbe7ffd63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:35:23.761420 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.761399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf6b61d-77c8-4610-8022-531bbe7ffd63-kube-api-access-8sb8k" (OuterVolumeSpecName: "kube-api-access-8sb8k") pod "cdf6b61d-77c8-4610-8022-531bbe7ffd63" (UID: "cdf6b61d-77c8-4610-8022-531bbe7ffd63"). InnerVolumeSpecName "kube-api-access-8sb8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:35:23.827216 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.827178    2580 generic.go:356] "Generic (PLEG): container finished" podID="cdf6b61d-77c8-4610-8022-531bbe7ffd63" containerID="5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1" exitCode=0
Sep 08 07:35:23.827386 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.827228    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"
Sep 08 07:35:23.827386 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.827271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h" event={"ID":"cdf6b61d-77c8-4610-8022-531bbe7ffd63","Type":"ContainerDied","Data":"5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1"}
Sep 08 07:35:23.827386 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.827323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h" event={"ID":"cdf6b61d-77c8-4610-8022-531bbe7ffd63","Type":"ContainerDied","Data":"6d744e4d67440272a312e4434f5224592c288c75ebaee363d2418f0a028c71a1"}
Sep 08 07:35:23.827386 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.827350    2580 scope.go:117] "RemoveContainer" containerID="5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1"
Sep 08 07:35:23.836656 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.836641    2580 scope.go:117] "RemoveContainer" containerID="5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1"
Sep 08 07:35:23.836910 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:35:23.836889    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1\": container with ID starting with 5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1 not found: ID does not exist" containerID="5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1"
Sep 08 07:35:23.836953 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.836918    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1"} err="failed to get container status \"5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1\": rpc error: code = NotFound desc = could not find container \"5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1\": container with ID starting with 5ab32db715333438bd5a1af580a9980a1023734458150e12cd39761a28177df1 not found: ID does not exist"
Sep 08 07:35:23.857919 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.857889    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"]
Sep 08 07:35:23.860502 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.860488    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6b61d-77c8-4610-8022-531bbe7ffd63-plugins-config-volume\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:23.860565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.860505    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sb8k\" (UniqueName: \"kubernetes.io/projected/cdf6b61d-77c8-4610-8022-531bbe7ffd63-kube-api-access-8sb8k\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:23.871518 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:23.871449    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fb59c2-77885bff6b-n9t8h"]
Sep 08 07:35:24.616829 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:24.616794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:24.940213 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:24.940124    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d094ea7-9a2d-45b0-ab2c-cf49c525374a" path="/var/lib/kubelet/pods/2d094ea7-9a2d-45b0-ab2c-cf49c525374a/volumes"
Sep 08 07:35:24.940537 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:24.940523    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdf6b61d-77c8-4610-8022-531bbe7ffd63" path="/var/lib/kubelet/pods/cdf6b61d-77c8-4610-8022-531bbe7ffd63/volumes"
Sep 08 07:35:28.700054 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:28.700015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:31.482079 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:31.482043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:31.544852 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:31.544797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:31.792244 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:31.792155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:32.199230 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.199193    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"]
Sep 08 07:35:32.199536 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.199522    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d094ea7-9a2d-45b0-ab2c-cf49c525374a" containerName="revoke-keys"
Sep 08 07:35:32.199599 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.199537    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d094ea7-9a2d-45b0-ab2c-cf49c525374a" containerName="revoke-keys"
Sep 08 07:35:32.199599 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.199564    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf6b61d-77c8-4610-8022-531bbe7ffd63" containerName="payload-pre-processing"
Sep 08 07:35:32.199599 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.199570    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf6b61d-77c8-4610-8022-531bbe7ffd63" containerName="payload-pre-processing"
Sep 08 07:35:32.199719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.199617    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf6b61d-77c8-4610-8022-531bbe7ffd63" containerName="payload-pre-processing"
Sep 08 07:35:32.199719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.199627    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d094ea7-9a2d-45b0-ab2c-cf49c525374a" containerName="revoke-keys"
Sep 08 07:35:32.202802 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.202786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"
Sep 08 07:35:32.205573 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.205551    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-824nh\""
Sep 08 07:35:32.206337 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.206317    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:35:32.216351 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.216328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"]
Sep 08 07:35:32.326632 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.326600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:32.342533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.342507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f46ce-9957-450f-b353-6e0462f16eb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf\" (UID: \"365f46ce-9957-450f-b353-6e0462f16eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"
Sep 08 07:35:32.443352 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.443315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f46ce-9957-450f-b353-6e0462f16eb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf\" (UID: \"365f46ce-9957-450f-b353-6e0462f16eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"
Sep 08 07:35:32.443933 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.443912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f46ce-9957-450f-b353-6e0462f16eb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf\" (UID: \"365f46ce-9957-450f-b353-6e0462f16eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"
Sep 08 07:35:32.513093 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.513012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"
Sep 08 07:35:32.789313 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.789287    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"]
Sep 08 07:35:32.791359 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:35:32.791331    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365f46ce_9957_450f_b353_6e0462f16eb2.slice/crio-b250f4588b37d0a8ef174bf3a992a0a5fa0630f25398c10d85744e6a7f8f233b WatchSource:0}: Error finding container b250f4588b37d0a8ef174bf3a992a0a5fa0630f25398c10d85744e6a7f8f233b: Status 404 returned error can't find the container with id b250f4588b37d0a8ef174bf3a992a0a5fa0630f25398c10d85744e6a7f8f233b
Sep 08 07:35:32.875005 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:32.874973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf" event={"ID":"365f46ce-9957-450f-b353-6e0462f16eb2","Type":"ContainerStarted","Data":"b250f4588b37d0a8ef174bf3a992a0a5fa0630f25398c10d85744e6a7f8f233b"}
Sep 08 07:35:33.882565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:33.882529    2580 generic.go:356] "Generic (PLEG): container finished" podID="365f46ce-9957-450f-b353-6e0462f16eb2" containerID="6f5279cb8699f175226b55f46b25e1b14c7dd45c67b840896bb7ac9d9e584ea5" exitCode=0
Sep 08 07:35:33.883020 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:33.882601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf" event={"ID":"365f46ce-9957-450f-b353-6e0462f16eb2","Type":"ContainerDied","Data":"6f5279cb8699f175226b55f46b25e1b14c7dd45c67b840896bb7ac9d9e584ea5"}
Sep 08 07:35:35.015991 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:35.015969    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"
Sep 08 07:35:35.168249 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:35.168162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f46ce-9957-450f-b353-6e0462f16eb2-maas-api-service-ca\") pod \"365f46ce-9957-450f-b353-6e0462f16eb2\" (UID: \"365f46ce-9957-450f-b353-6e0462f16eb2\") "
Sep 08 07:35:35.168533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:35.168512    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365f46ce-9957-450f-b353-6e0462f16eb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "365f46ce-9957-450f-b353-6e0462f16eb2" (UID: "365f46ce-9957-450f-b353-6e0462f16eb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:35:35.269436 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:35.269401    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f46ce-9957-450f-b353-6e0462f16eb2-maas-api-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:35:35.893305 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:35.893277    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"
Sep 08 07:35:35.893497 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:35.893276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf" event={"ID":"365f46ce-9957-450f-b353-6e0462f16eb2","Type":"ContainerDied","Data":"b250f4588b37d0a8ef174bf3a992a0a5fa0630f25398c10d85744e6a7f8f233b"}
Sep 08 07:35:35.893497 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:35.893411    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b250f4588b37d0a8ef174bf3a992a0a5fa0630f25398c10d85744e6a7f8f233b"
Sep 08 07:35:42.828347 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:42.828306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:42.843894 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:42.843857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"]
Sep 08 07:35:42.848365 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:42.848339    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-217bc2-29814210-dn4nx"]
Sep 08 07:35:42.927345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:42.927303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:42.940029 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:42.939998    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a6c537-6806-4fef-b5e2-6d7965ced02c" path="/var/lib/kubelet/pods/41a6c537-6806-4fef-b5e2-6d7965ced02c/volumes"
Sep 08 07:35:43.030258 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:43.030227    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:43.125597 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:43.125517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:43.138856 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:43.138828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"]
Sep 08 07:35:43.143661 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:43.143635    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-217bc2-zh7kf"]
Sep 08 07:35:43.187070 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:43.187042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:43.513438 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:43.513345    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:44.941137 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:44.941107    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="365f46ce-9957-450f-b353-6e0462f16eb2" path="/var/lib/kubelet/pods/365f46ce-9957-450f-b353-6e0462f16eb2/volumes"
Sep 08 07:35:45.176012 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:45.175982    2580 scope.go:117] "RemoveContainer" containerID="d46e2f0e7190f510ceae95cbff8c1bac3f12a89d5811c62b07461c05a5cd56db"
Sep 08 07:35:51.492435 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:51.492402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:51.562525 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:51.562494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:35:51.818998 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:35:51.818967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:36:31.179874 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:31.179835    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:36:45.210035 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:45.210005    2580 scope.go:117] "RemoveContainer" containerID="35d72fc6a355e5bfe805a03dec2cbefb7e9bf9688ea81b6f04c64f7addd1684e"
Sep 08 07:36:45.220497 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:45.220453    2580 scope.go:117] "RemoveContainer" containerID="3e1511922c3799ccf8cfd1867c590d6b5212d0d2fd0e9f43e8d515bc25a5ffeb"
Sep 08 07:36:45.228010 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:45.227989    2580 scope.go:117] "RemoveContainer" containerID="a00bea219fc74f575f2017960b8bcf5e077742d2799e98ab2df5960be729f4d0"
Sep 08 07:36:45.235101 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:45.235084    2580 scope.go:117] "RemoveContainer" containerID="d8435a4cde317a9802bab1f614a6055186454eeeb297c72d12d79b968041d95d"
Sep 08 07:36:45.242129 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:45.242112    2580 scope.go:117] "RemoveContainer" containerID="bd5f7bb3932313cfe0afa0a139eb1ce62d527ba2a720581718c76428aa9b1006"
Sep 08 07:36:45.248894 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:45.248867    2580 scope.go:117] "RemoveContainer" containerID="fb6c5e393a91d3e2a899acff69172e4d2f639b631554fb3dc1a0e39cccb7edd1"
Sep 08 07:36:49.772858 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:49.772822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:36:49.830306 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:36:49.830277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:37:14.704584 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:37:14.704538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:37:15.036500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:37:15.036399    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:37:16.855584 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:37:16.855534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:38:45.116268 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:38:45.116229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:38:45.177158 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:38:45.177124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:39:59.252571 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:39:59.252532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:40:12.645241 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.645205    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"]
Sep 08 07:40:12.647908 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.645570    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365f46ce-9957-450f-b353-6e0462f16eb2" containerName="revoke-keys"
Sep 08 07:40:12.647908 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.645584    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="365f46ce-9957-450f-b353-6e0462f16eb2" containerName="revoke-keys"
Sep 08 07:40:12.647908 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.645662    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="365f46ce-9957-450f-b353-6e0462f16eb2" containerName="revoke-keys"
Sep 08 07:40:12.648757 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.648741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.650629 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.650606    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-220f0417-kserve-self-signed-certs\""
Sep 08 07:40:12.655818 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.655787    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"]
Sep 08 07:40:12.669827 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.669799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:40:12.763372 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.763333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57d4r\" (UniqueName: \"kubernetes.io/projected/c31de66e-b400-44f2-bb7f-fecd8bb9add9-kube-api-access-57d4r\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.763609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.763403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tmp-dir\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.763609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.763513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-home\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.763609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.763593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-model-cache\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.763733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.763614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-dshm\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.763733 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.763636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tls-certs\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864383 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-home\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-model-cache\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-dshm\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tls-certs\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57d4r\" (UniqueName: \"kubernetes.io/projected/c31de66e-b400-44f2-bb7f-fecd8bb9add9-kube-api-access-57d4r\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864586 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tmp-dir\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864854 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-home\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864909 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-model-cache\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.864952 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.864932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tmp-dir\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.866826 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.866802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-dshm\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.867015 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.866993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tls-certs\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.872627 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.872604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57d4r\" (UniqueName: \"kubernetes.io/projected/c31de66e-b400-44f2-bb7f-fecd8bb9add9-kube-api-access-57d4r\") pod \"e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") " pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:12.959360 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:12.959267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:13.232290 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:13.232209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"]
Sep 08 07:40:13.235857 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:40:13.235832    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc31de66e_b400_44f2_bb7f_fecd8bb9add9.slice/crio-30b5da72d99c13ad446741910f5fba0cb3850d662405b07a072707d9e93e09af WatchSource:0}: Error finding container 30b5da72d99c13ad446741910f5fba0cb3850d662405b07a072707d9e93e09af: Status 404 returned error can't find the container with id 30b5da72d99c13ad446741910f5fba0cb3850d662405b07a072707d9e93e09af
Sep 08 07:40:14.207178 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:14.207135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c" event={"ID":"c31de66e-b400-44f2-bb7f-fecd8bb9add9","Type":"ContainerStarted","Data":"ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b"}
Sep 08 07:40:14.207178 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:14.207181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c" event={"ID":"c31de66e-b400-44f2-bb7f-fecd8bb9add9","Type":"ContainerStarted","Data":"30b5da72d99c13ad446741910f5fba0cb3850d662405b07a072707d9e93e09af"}
Sep 08 07:40:14.224897 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:14.224834    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c" podStartSLOduration=2.224819841 podStartE2EDuration="2.224819841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:40:14.222179427 +0000 UTC m=+2129.905639579" watchObservedRunningTime="2026-09-08 07:40:14.224819841 +0000 UTC m=+2129.908279993"
Sep 08 07:40:22.960114 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:22.960045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:22.960114 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:22.960129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:22.972819 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:22.972783    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:23.267276 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:23.267187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:40:50.445771 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:50.445740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:40:52.127624 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:52.127591    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"]
Sep 08 07:40:52.147534 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:40:52.147504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:41:07.147145 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.147078    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c" podUID="c31de66e-b400-44f2-bb7f-fecd8bb9add9" containerName="main" containerID="cri-o://ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b" gracePeriod=45
Sep 08 07:41:07.387533 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.387497    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:41:07.462422 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.462337    2580 generic.go:356] "Generic (PLEG): container finished" podID="c31de66e-b400-44f2-bb7f-fecd8bb9add9" containerID="ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b" exitCode=0
Sep 08 07:41:07.462422 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.462380    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"
Sep 08 07:41:07.462656 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.462420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c" event={"ID":"c31de66e-b400-44f2-bb7f-fecd8bb9add9","Type":"ContainerDied","Data":"ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b"}
Sep 08 07:41:07.462656 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.462480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c" event={"ID":"c31de66e-b400-44f2-bb7f-fecd8bb9add9","Type":"ContainerDied","Data":"30b5da72d99c13ad446741910f5fba0cb3850d662405b07a072707d9e93e09af"}
Sep 08 07:41:07.462656 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.462497    2580 scope.go:117] "RemoveContainer" containerID="ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b"
Sep 08 07:41:07.470787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.470772    2580 scope.go:117] "RemoveContainer" containerID="ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b"
Sep 08 07:41:07.471067 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:41:07.471050    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b\": container with ID starting with ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b not found: ID does not exist" containerID="ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b"
Sep 08 07:41:07.471120 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.471076    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b"} err="failed to get container status \"ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b\": rpc error: code = NotFound desc = could not find container \"ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b\": container with ID starting with ca125bbb82d483a86072c0518b35d8716935107038ae7a575749d1431b43f68b not found: ID does not exist"
Sep 08 07:41:07.479356 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tmp-dir\") pod \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") "
Sep 08 07:41:07.479452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-dshm\") pod \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") "
Sep 08 07:41:07.479452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-model-cache\") pod \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") "
Sep 08 07:41:07.479600 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479508    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tls-certs\") pod \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") "
Sep 08 07:41:07.479600 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479545    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57d4r\" (UniqueName: \"kubernetes.io/projected/c31de66e-b400-44f2-bb7f-fecd8bb9add9-kube-api-access-57d4r\") pod \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") "
Sep 08 07:41:07.479600 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c31de66e-b400-44f2-bb7f-fecd8bb9add9" (UID: "c31de66e-b400-44f2-bb7f-fecd8bb9add9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:41:07.479600 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-home\") pod \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\" (UID: \"c31de66e-b400-44f2-bb7f-fecd8bb9add9\") "
Sep 08 07:41:07.479787 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c31de66e-b400-44f2-bb7f-fecd8bb9add9" (UID: "c31de66e-b400-44f2-bb7f-fecd8bb9add9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:41:07.479940 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479913    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-home" (OuterVolumeSpecName: "home") pod "c31de66e-b400-44f2-bb7f-fecd8bb9add9" (UID: "c31de66e-b400-44f2-bb7f-fecd8bb9add9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:41:07.480054 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479979    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tmp-dir\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:07.480054 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.479995    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-model-cache\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:07.481609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.481583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-dshm" (OuterVolumeSpecName: "dshm") pod "c31de66e-b400-44f2-bb7f-fecd8bb9add9" (UID: "c31de66e-b400-44f2-bb7f-fecd8bb9add9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 07:41:07.481697 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.481602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c31de66e-b400-44f2-bb7f-fecd8bb9add9" (UID: "c31de66e-b400-44f2-bb7f-fecd8bb9add9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:41:07.481697 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.481664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31de66e-b400-44f2-bb7f-fecd8bb9add9-kube-api-access-57d4r" (OuterVolumeSpecName: "kube-api-access-57d4r") pod "c31de66e-b400-44f2-bb7f-fecd8bb9add9" (UID: "c31de66e-b400-44f2-bb7f-fecd8bb9add9"). InnerVolumeSpecName "kube-api-access-57d4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:41:07.581087 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.581043    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-dshm\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:07.581087 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.581079    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31de66e-b400-44f2-bb7f-fecd8bb9add9-tls-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:07.581087 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.581094    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57d4r\" (UniqueName: \"kubernetes.io/projected/c31de66e-b400-44f2-bb7f-fecd8bb9add9-kube-api-access-57d4r\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:07.581356 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.581106    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31de66e-b400-44f2-bb7f-fecd8bb9add9-home\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:07.784887 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.784633    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"]
Sep 08 07:41:07.789811 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:07.789773    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-220f0417-kserve-6578ddcf87-9hf2c"]
Sep 08 07:41:08.940265 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:08.940228    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31de66e-b400-44f2-bb7f-fecd8bb9add9" path="/var/lib/kubelet/pods/c31de66e-b400-44f2-bb7f-fecd8bb9add9/volumes"
Sep 08 07:41:15.347650 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.347616    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55f944c447-x84ss"]
Sep 08 07:41:15.348044 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.347951    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31de66e-b400-44f2-bb7f-fecd8bb9add9" containerName="main"
Sep 08 07:41:15.348044 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.347962    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31de66e-b400-44f2-bb7f-fecd8bb9add9" containerName="main"
Sep 08 07:41:15.348044 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.348029    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31de66e-b400-44f2-bb7f-fecd8bb9add9" containerName="main"
Sep 08 07:41:15.352310 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.352288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.354931 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.354906    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 07:41:15.355147 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.355130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qfrsm\""
Sep 08 07:41:15.355236 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.355144    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 07:41:15.355236 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.355167    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 07:41:15.355356 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.355241    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 07:41:15.360609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.360140    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f944c447-x84ss"]
Sep 08 07:41:15.450055 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.450018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14527a52-bd58-40fc-aaa1-e1c8dd80b641-openshift-service-ca\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.450055 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.450055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs297\" (UniqueName: \"kubernetes.io/projected/14527a52-bd58-40fc-aaa1-e1c8dd80b641-kube-api-access-bs297\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.450258 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.450075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-metrics-certs\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.450258 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.450094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-maas-api-tls\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.551026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.550990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14527a52-bd58-40fc-aaa1-e1c8dd80b641-openshift-service-ca\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.551026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.551024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs297\" (UniqueName: \"kubernetes.io/projected/14527a52-bd58-40fc-aaa1-e1c8dd80b641-kube-api-access-bs297\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.551238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.551043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-metrics-certs\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.551238 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.551068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-maas-api-tls\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.551709 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.551685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14527a52-bd58-40fc-aaa1-e1c8dd80b641-openshift-service-ca\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.553610 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.553582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-maas-api-tls\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.553705 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.553640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-metrics-certs\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.558185 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.558169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs297\" (UniqueName: \"kubernetes.io/projected/14527a52-bd58-40fc-aaa1-e1c8dd80b641-kube-api-access-bs297\") pod \"maas-api-55f944c447-x84ss\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") " pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.664452 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.664360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:15.935105 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.935078    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f944c447-x84ss"]
Sep 08 07:41:15.937223 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:41:15.937197    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14527a52_bd58_40fc_aaa1_e1c8dd80b641.slice/crio-50d75644dc40a5586a590fafa5b65c2a7dd8d4e9c2a9bf2eed3c43d13aa5c159 WatchSource:0}: Error finding container 50d75644dc40a5586a590fafa5b65c2a7dd8d4e9c2a9bf2eed3c43d13aa5c159: Status 404 returned error can't find the container with id 50d75644dc40a5586a590fafa5b65c2a7dd8d4e9c2a9bf2eed3c43d13aa5c159
Sep 08 07:41:15.938670 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:15.938653    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 07:41:16.505375 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:16.505336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss" event={"ID":"14527a52-bd58-40fc-aaa1-e1c8dd80b641","Type":"ContainerStarted","Data":"50d75644dc40a5586a590fafa5b65c2a7dd8d4e9c2a9bf2eed3c43d13aa5c159"}
Sep 08 07:41:17.511691 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:17.511649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss" event={"ID":"14527a52-bd58-40fc-aaa1-e1c8dd80b641","Type":"ContainerStarted","Data":"2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19"}
Sep 08 07:41:17.512127 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:17.511728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:17.527719 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:17.527646    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss" podStartSLOduration=1.611794934 podStartE2EDuration="2.527627409s" totalImagesPullingTime="915.832475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:41:17.526274944 +0000 UTC m=+2193.209735097" watchObservedRunningTime="2026-09-08 07:41:17.527627409 +0000 UTC m=+2193.211087563"
Sep 08 07:41:23.254622 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.254580    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f944c447-x84ss"]
Sep 08 07:41:23.255146 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.254769    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss" podUID="14527a52-bd58-40fc-aaa1-e1c8dd80b641" containerName="maas-api" containerID="cri-o://2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19" gracePeriod=30
Sep 08 07:41:23.260008 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.259983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:23.284159 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.284131    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b546674f9-nntfb"]
Sep 08 07:41:23.287361 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.287343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.300879 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.300841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b546674f9-nntfb"]
Sep 08 07:41:23.428083 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.428052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfx4x\" (UniqueName: \"kubernetes.io/projected/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-kube-api-access-bfx4x\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.428248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.428125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-maas-api-tls\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.428248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.428161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-metrics-certs\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.428248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.428197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-openshift-service-ca\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.499943 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.499917    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:23.529293 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.529205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-maas-api-tls\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.529293 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.529241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-metrics-certs\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.529293 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.529279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-openshift-service-ca\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.529593 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.529332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfx4x\" (UniqueName: \"kubernetes.io/projected/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-kube-api-access-bfx4x\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.530015 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.529988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-openshift-service-ca\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.531728 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.531695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-maas-api-tls\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.531851 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.531794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-metrics-certs\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.537653 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.537623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfx4x\" (UniqueName: \"kubernetes.io/projected/b60882f3-75a0-4aab-aac0-1b7dbea6f1a4-kube-api-access-bfx4x\") pod \"maas-api-b546674f9-nntfb\" (UID: \"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4\") " pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.541623 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.541570    2580 generic.go:356] "Generic (PLEG): container finished" podID="14527a52-bd58-40fc-aaa1-e1c8dd80b641" containerID="2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19" exitCode=0
Sep 08 07:41:23.541786 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.541635    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss"
Sep 08 07:41:23.541786 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.541646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss" event={"ID":"14527a52-bd58-40fc-aaa1-e1c8dd80b641","Type":"ContainerDied","Data":"2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19"}
Sep 08 07:41:23.541786 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.541749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f944c447-x84ss" event={"ID":"14527a52-bd58-40fc-aaa1-e1c8dd80b641","Type":"ContainerDied","Data":"50d75644dc40a5586a590fafa5b65c2a7dd8d4e9c2a9bf2eed3c43d13aa5c159"}
Sep 08 07:41:23.541976 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.541809    2580 scope.go:117] "RemoveContainer" containerID="2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19"
Sep 08 07:41:23.553809 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.553784    2580 scope.go:117] "RemoveContainer" containerID="2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19"
Sep 08 07:41:23.554129 ip-10-0-139-109 kubenswrapper[2580]: E0908 07:41:23.554110    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19\": container with ID starting with 2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19 not found: ID does not exist" containerID="2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19"
Sep 08 07:41:23.554182 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.554138    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19"} err="failed to get container status \"2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19\": rpc error: code = NotFound desc = could not find container \"2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19\": container with ID starting with 2e3679d5aecd00075bcb6f86793cd90ba99adaf47f94f8bcf05ab385e6716f19 not found: ID does not exist"
Sep 08 07:41:23.630216 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.630176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs297\" (UniqueName: \"kubernetes.io/projected/14527a52-bd58-40fc-aaa1-e1c8dd80b641-kube-api-access-bs297\") pod \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") "
Sep 08 07:41:23.630388 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.630247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14527a52-bd58-40fc-aaa1-e1c8dd80b641-openshift-service-ca\") pod \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") "
Sep 08 07:41:23.630388 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.630328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-maas-api-tls\") pod \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") "
Sep 08 07:41:23.630388 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.630345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-metrics-certs\") pod \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\" (UID: \"14527a52-bd58-40fc-aaa1-e1c8dd80b641\") "
Sep 08 07:41:23.630661 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.630631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14527a52-bd58-40fc-aaa1-e1c8dd80b641-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14527a52-bd58-40fc-aaa1-e1c8dd80b641" (UID: "14527a52-bd58-40fc-aaa1-e1c8dd80b641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 07:41:23.632421 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.632389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "14527a52-bd58-40fc-aaa1-e1c8dd80b641" (UID: "14527a52-bd58-40fc-aaa1-e1c8dd80b641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:41:23.632421 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.632403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14527a52-bd58-40fc-aaa1-e1c8dd80b641" (UID: "14527a52-bd58-40fc-aaa1-e1c8dd80b641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 07:41:23.632614 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.632399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14527a52-bd58-40fc-aaa1-e1c8dd80b641-kube-api-access-bs297" (OuterVolumeSpecName: "kube-api-access-bs297") pod "14527a52-bd58-40fc-aaa1-e1c8dd80b641" (UID: "14527a52-bd58-40fc-aaa1-e1c8dd80b641"). InnerVolumeSpecName "kube-api-access-bs297". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 07:41:23.637609 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.637590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:23.731159 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.731126    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs297\" (UniqueName: \"kubernetes.io/projected/14527a52-bd58-40fc-aaa1-e1c8dd80b641-kube-api-access-bs297\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:23.731159 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.731154    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14527a52-bd58-40fc-aaa1-e1c8dd80b641-openshift-service-ca\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:23.731159 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.731164    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-maas-api-tls\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:23.731393 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.731176    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14527a52-bd58-40fc-aaa1-e1c8dd80b641-metrics-certs\") on node \"ip-10-0-139-109.ec2.internal\" DevicePath \"\""
Sep 08 07:41:23.863760 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.863722    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f944c447-x84ss"]
Sep 08 07:41:23.865275 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.865254    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f944c447-x84ss"]
Sep 08 07:41:23.908045 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:23.907856    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b546674f9-nntfb"]
Sep 08 07:41:23.910166 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:41:23.910139    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60882f3_75a0_4aab_aac0_1b7dbea6f1a4.slice/crio-4e9c5338557ce328d745c4858a307e9caef13fc5becc5a205e21259c1975ef60 WatchSource:0}: Error finding container 4e9c5338557ce328d745c4858a307e9caef13fc5becc5a205e21259c1975ef60: Status 404 returned error can't find the container with id 4e9c5338557ce328d745c4858a307e9caef13fc5becc5a205e21259c1975ef60
Sep 08 07:41:24.547541 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:24.547429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb" event={"ID":"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4","Type":"ContainerStarted","Data":"a2385de785320b8f89fd5c63713472273d8c87e0f4cb9e5a4e6471fbf427dc4d"}
Sep 08 07:41:24.547919 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:24.547614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:24.547919 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:24.547656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb" event={"ID":"b60882f3-75a0-4aab-aac0-1b7dbea6f1a4","Type":"ContainerStarted","Data":"4e9c5338557ce328d745c4858a307e9caef13fc5becc5a205e21259c1975ef60"}
Sep 08 07:41:24.565006 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:24.564957    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb" podStartSLOduration=1.229693168 podStartE2EDuration="1.564941951s" totalImagesPullingTime="335.248783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 07:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:41:24.562912579 +0000 UTC m=+2200.246372736" watchObservedRunningTime="2026-09-08 07:41:24.564941951 +0000 UTC m=+2200.248402103"
Sep 08 07:41:24.940590 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:24.940561    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14527a52-bd58-40fc-aaa1-e1c8dd80b641" path="/var/lib/kubelet/pods/14527a52-bd58-40fc-aaa1-e1c8dd80b641/volumes"
Sep 08 07:41:30.559649 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:30.559608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b546674f9-nntfb"
Sep 08 07:41:45.026345 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:45.022144    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-648b45f788-rjlcc_87260785-3347-4707-b458-fc07bbcc8104/authorino/0.log"
Sep 08 07:41:45.309807 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:45.309727    2580 scope.go:117] "RemoveContainer" containerID="9e9d244e2cc4e62915b37b8f364cb9ecdf14107affeb183f68df2f0adcc3e113"
Sep 08 07:41:45.317816 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:45.317789    2580 scope.go:117] "RemoveContainer" containerID="6f5279cb8699f175226b55f46b25e1b14c7dd45c67b840896bb7ac9d9e584ea5"
Sep 08 07:41:50.931945 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:50.931914    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79c9ffb96f-cqzk9_557dcd52-2243-4aef-8041-1fcb15748061/copy-manifests/0.log"
Sep 08 07:41:50.939342 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:50.939313    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79c9ffb96f-cqzk9_557dcd52-2243-4aef-8041-1fcb15748061/manager/0.log"
Sep 08 07:41:51.379916 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:51.379885    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b546674f9-nntfb_b60882f3-75a0-4aab-aac0-1b7dbea6f1a4/maas-api/0.log"
Sep 08 07:41:52.431936 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:52.431902    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-648b45f788-rjlcc_87260785-3347-4707-b458-fc07bbcc8104/authorino/0.log"
Sep 08 07:41:52.752526 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:52.752428    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-pg7jc_920e6331-fed7-4317-b3a6-05f6bd6032a2/kuadrant-console-plugin/0.log"
Sep 08 07:41:53.091326 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:53.091295    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-8hv5c_4a5651b3-8356-48ef-a4fe-759712a192c7/manager/0.log"
Sep 08 07:41:53.763584 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:53.763552    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd6874cf-t744g_c39bc3c3-e554-4582-8949-da95477c5b82/kube-auth-proxy/0.log"
Sep 08 07:41:54.530138 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:54.530107    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9nsh9_e01955d1-bdf8-41c7-a042-e1bbeb7af050/main/0.log"
Sep 08 07:41:54.852258 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:54.852228    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-c97v4_829d1e3c-c13d-466a-8a04-175993c07f8d/main/0.log"
Sep 08 07:41:55.070307 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:55.070278    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-7ms5d_c61fcf99-aff9-4df8-9fd3-28c34619a7e5/main/0.log"
Sep 08 07:41:55.174202 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:41:55.174125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-75ggk_efb705ec-c8b9-49f7-95b7-dac2f6002e26/main/0.log"
Sep 08 07:42:00.157711 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:00.157676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:42:00.223595 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:00.223555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:42:04.604881 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:04.604847    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zrqsh_bb1a056e-d8cc-4090-96ef-c2f53cfa99a8/global-pull-secret-syncer/0.log"
Sep 08 07:42:04.650350 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:04.650320    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log"
Sep 08 07:42:04.755146 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:04.755115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-109.ec2.internal_901888addc1cc153e7772482f8ca35e4/haproxy/0.log"
Sep 08 07:42:08.026341 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:08.026253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6g8t9_40e80d3a-bc71-4e0b-8616-f022edcc6fcb/konnectivity-agent/0.log" limit=1024
Sep 08 07:42:09.061931 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:09.061896    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-648b45f788-rjlcc_87260785-3347-4707-b458-fc07bbcc8104/authorino/0.log"
Sep 08 07:42:09.178294 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:09.178261    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-pg7jc_920e6331-fed7-4317-b3a6-05f6bd6032a2/kuadrant-console-plugin/0.log"
Sep 08 07:42:09.303920 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:09.303890    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-8hv5c_4a5651b3-8356-48ef-a4fe-759712a192c7/manager/0.log"
Sep 08 07:42:11.054289 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.054217    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9d98b4687-2grc6_7ef724af-a951-4c35-aba2-08cf0d9b0b0f/metrics-server/0.log"
Sep 08 07:42:11.280686 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.280657    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vm878_fc4e3f3b-6bd3-4752-82e3-a373c44729b5/node-exporter/0.log"
Sep 08 07:42:11.302500 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.302474    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vm878_fc4e3f3b-6bd3-4752-82e3-a373c44729b5/kube-rbac-proxy/0.log"
Sep 08 07:42:11.324566 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.324543    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vm878_fc4e3f3b-6bd3-4752-82e3-a373c44729b5/init-textfile/0.log"
Sep 08 07:42:11.439349 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.439324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2583030-8f67-49b4-8b2b-5f2e08a006ed/prometheus/0.log"
Sep 08 07:42:11.461508 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.461486    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2583030-8f67-49b4-8b2b-5f2e08a006ed/config-reloader/0.log"
Sep 08 07:42:11.499353 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.499324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2583030-8f67-49b4-8b2b-5f2e08a006ed/thanos-sidecar/0.log"
Sep 08 07:42:11.522174 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.522150    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2583030-8f67-49b4-8b2b-5f2e08a006ed/kube-rbac-proxy-web/0.log"
Sep 08 07:42:11.548260 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.548236    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2583030-8f67-49b4-8b2b-5f2e08a006ed/kube-rbac-proxy/0.log"
Sep 08 07:42:11.571058 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.571036    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2583030-8f67-49b4-8b2b-5f2e08a006ed/kube-rbac-proxy-thanos/0.log"
Sep 08 07:42:11.607910 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.607887    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2583030-8f67-49b4-8b2b-5f2e08a006ed/init-config-reloader/0.log"
Sep 08 07:42:11.686165 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.686139    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-l5gks_f1f496c7-6c94-4477-895f-90f5f2ff589d/prometheus-operator/0.log"
Sep 08 07:42:11.708542 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.708518    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-l5gks_f1f496c7-6c94-4477-895f-90f5f2ff589d/kube-rbac-proxy/0.log"
Sep 08 07:42:11.863781 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.863692    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6d664c87-2qd5d_040bedf5-aaec-4274-8338-1b97faf7dca3/thanos-query/0.log"
Sep 08 07:42:11.887849 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.887824    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6d664c87-2qd5d_040bedf5-aaec-4274-8338-1b97faf7dca3/kube-rbac-proxy-web/0.log"
Sep 08 07:42:11.915572 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.915551    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6d664c87-2qd5d_040bedf5-aaec-4274-8338-1b97faf7dca3/kube-rbac-proxy/0.log"
Sep 08 07:42:11.939949 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.939928    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6d664c87-2qd5d_040bedf5-aaec-4274-8338-1b97faf7dca3/prom-label-proxy/0.log"
Sep 08 07:42:11.963183 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.963163    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6d664c87-2qd5d_040bedf5-aaec-4274-8338-1b97faf7dca3/kube-rbac-proxy-rules/0.log"
Sep 08 07:42:11.990475 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:11.990436    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6d664c87-2qd5d_040bedf5-aaec-4274-8338-1b97faf7dca3/kube-rbac-proxy-metrics/0.log"
Sep 08 07:42:14.650293 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.650262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-768697c4b8-wdwdt_8f4df6fa-46c4-406b-9f2e-bf53c32233da/console/0.log"
Sep 08 07:42:14.771222 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.771179    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"]
Sep 08 07:42:14.771716 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.771690    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14527a52-bd58-40fc-aaa1-e1c8dd80b641" containerName="maas-api"
Sep 08 07:42:14.771716 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.771716    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="14527a52-bd58-40fc-aaa1-e1c8dd80b641" containerName="maas-api"
Sep 08 07:42:14.771886 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.771779    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="14527a52-bd58-40fc-aaa1-e1c8dd80b641" containerName="maas-api"
Sep 08 07:42:14.775149 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.775128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.777069 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.777046    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jg2tn\"/\"default-dockercfg-jjph5\""
Sep 08 07:42:14.777245 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.777223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jg2tn\"/\"openshift-service-ca.crt\""
Sep 08 07:42:14.777720 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.777707    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jg2tn\"/\"kube-root-ca.crt\""
Sep 08 07:42:14.784409 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.784386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"]
Sep 08 07:42:14.797026 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.796996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-lib-modules\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.797137 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.797077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-proc\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.797180 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.797135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-podres\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.797180 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.797160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-sys\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.797248 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.797206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bng2q\" (UniqueName: \"kubernetes.io/projected/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-kube-api-access-bng2q\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898340 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bng2q\" (UniqueName: \"kubernetes.io/projected/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-kube-api-access-bng2q\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-lib-modules\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-proc\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-podres\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-sys\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898565 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-proc\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898807 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-sys\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898807 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-lib-modules\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.898807 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.898592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-podres\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:14.905808 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:14.905742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bng2q\" (UniqueName: \"kubernetes.io/projected/9c68d030-0cb1-40d6-9212-2ae60c7dbe26-kube-api-access-bng2q\") pod \"perf-node-gather-daemonset-576qn\" (UID: \"9c68d030-0cb1-40d6-9212-2ae60c7dbe26\") " pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:15.087087 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:15.087052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:15.358931 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:15.358905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"]
Sep 08 07:42:15.361547 ip-10-0-139-109 kubenswrapper[2580]: W0908 07:42:15.361512    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9c68d030_0cb1_40d6_9212_2ae60c7dbe26.slice/crio-1877c0079feda2ba3c12125bfc50792d91c61a89873f9db46833a91d168b9aa7 WatchSource:0}: Error finding container 1877c0079feda2ba3c12125bfc50792d91c61a89873f9db46833a91d168b9aa7: Status 404 returned error can't find the container with id 1877c0079feda2ba3c12125bfc50792d91c61a89873f9db46833a91d168b9aa7
Sep 08 07:42:15.789950 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:15.789856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn" event={"ID":"9c68d030-0cb1-40d6-9212-2ae60c7dbe26","Type":"ContainerStarted","Data":"289c535d79a428d2ca163d762a8837231dec488c1ded0d6c1024ac013518ef49"}
Sep 08 07:42:15.789950 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:15.789920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn" event={"ID":"9c68d030-0cb1-40d6-9212-2ae60c7dbe26","Type":"ContainerStarted","Data":"1877c0079feda2ba3c12125bfc50792d91c61a89873f9db46833a91d168b9aa7"}
Sep 08 07:42:15.790354 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:15.789958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:15.807139 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:15.807083    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn" podStartSLOduration=1.807067787 podStartE2EDuration="1.807067787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 07:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 07:42:15.805103442 +0000 UTC m=+2251.488563595" watchObservedRunningTime="2026-09-08 07:42:15.807067787 +0000 UTC m=+2251.490527979"
Sep 08 07:42:16.683007 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:16.682977    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q8rpz_e7f79343-7c0b-4029-a54e-d73b1b5e7007/dns/0.log"
Sep 08 07:42:16.705582 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:16.705537    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q8rpz_e7f79343-7c0b-4029-a54e-d73b1b5e7007/kube-rbac-proxy/0.log"
Sep 08 07:42:16.817212 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:16.817185    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v95nw_d745d326-a791-4108-b6bd-59caff1b804b/dns-node-resolver/0.log"
Sep 08 07:42:17.404311 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:17.404279    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lc7d6_3e4a766e-393c-42b7-912b-5aabacc745f4/node-ca/0.log"
Sep 08 07:42:18.692320 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:18.692284    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd6874cf-t744g_c39bc3c3-e554-4582-8949-da95477c5b82/kube-auth-proxy/0.log"
Sep 08 07:42:19.689968 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:19.689942    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6t76l_e2c45c7a-3c98-48b3-aba0-e7da22438105/serve-healthcheck-canary/0.log"
Sep 08 07:42:20.349671 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:20.349641    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rczlb_0347d928-62cf-4ad9-8bdf-615920a4ee9e/kube-rbac-proxy/0.log"
Sep 08 07:42:20.372072 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:20.372045    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rczlb_0347d928-62cf-4ad9-8bdf-615920a4ee9e/exporter/0.log"
Sep 08 07:42:20.393124 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:20.393091    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rczlb_0347d928-62cf-4ad9-8bdf-615920a4ee9e/extractor/0.log"
Sep 08 07:42:21.805228 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:21.805200    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jg2tn/perf-node-gather-daemonset-576qn"
Sep 08 07:42:22.905125 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:22.905077    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79c9ffb96f-cqzk9_557dcd52-2243-4aef-8041-1fcb15748061/manager/0.log"
Sep 08 07:42:22.925896 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:22.925874    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79c9ffb96f-cqzk9_557dcd52-2243-4aef-8041-1fcb15748061/copy-manifests/0.log"
Sep 08 07:42:24.467139 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:24.467065    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-767bfdf788-qhfpp_5b9acb70-e8f3-4f55-aa22-6e16b3b0ac0d/manager/0.log"
Sep 08 07:42:24.493309 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:24.493275    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-n2jdc_c6cc4024-37dd-4bd0-a844-b88bd7f6999a/openshift-lws-operator/0.log"
Sep 08 07:42:25.859263 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:25.859235    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd6874cf-t744g_c39bc3c3-e554-4582-8949-da95477c5b82/kube-auth-proxy/0.log"
Sep 08 07:42:27.821303 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:27.821272    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cd6874cf-t744g_c39bc3c3-e554-4582-8949-da95477c5b82/kube-auth-proxy/0.log"
Sep 08 07:42:31.318443 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.318420    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8nvwv_37fef4ff-7a57-420d-ae7b-0dfd37943b3f/kube-multus-additional-cni-plugins/0.log"
Sep 08 07:42:31.341945 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.341921    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8nvwv_37fef4ff-7a57-420d-ae7b-0dfd37943b3f/egress-router-binary-copy/0.log"
Sep 08 07:42:31.364030 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.363997    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8nvwv_37fef4ff-7a57-420d-ae7b-0dfd37943b3f/cni-plugins/0.log"
Sep 08 07:42:31.388813 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.388794    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8nvwv_37fef4ff-7a57-420d-ae7b-0dfd37943b3f/bond-cni-plugin/0.log"
Sep 08 07:42:31.410099 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.410078    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8nvwv_37fef4ff-7a57-420d-ae7b-0dfd37943b3f/routeoverride-cni/0.log"
Sep 08 07:42:31.435260 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.435239    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8nvwv_37fef4ff-7a57-420d-ae7b-0dfd37943b3f/whereabouts-cni-bincopy/0.log"
Sep 08 07:42:31.457030 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.457009    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8nvwv_37fef4ff-7a57-420d-ae7b-0dfd37943b3f/whereabouts-cni/0.log"
Sep 08 07:42:31.854055 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.854025    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qwkps_8b7a2c1b-1105-437c-bdca-c0e6495ff408/kube-multus/0.log"
Sep 08 07:42:31.934604 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.934575    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hb6qs_18397989-832c-4e4d-b309-f09e6c5e0207/network-metrics-daemon/0.log"
Sep 08 07:42:31.953119 ip-10-0-139-109 kubenswrapper[2580]: I0908 07:42:31.953095    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hb6qs_18397989-832c-4e4d-b309-f09e6c5e0207/kube-rbac-proxy/0.log"