Sep 10 23:36:28.055195 ip-10-0-137-218 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 23:36:28.444181 ip-10-0-137-218 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:36:28.444181 ip-10-0-137-218 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 23:36:28.444181 ip-10-0-137-218 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:36:28.444181 ip-10-0-137-218 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 23:36:28.444181 ip-10-0-137-218 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:36:28.445902 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.445790    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 23:36:28.449462 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449446    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:28.449462 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449462    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449467    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449471    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449474    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449477    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449480    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449482    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449485    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449488    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449491    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449493    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449496    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449500    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449504    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449507    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449509    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449512    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449514    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449517    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449519    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:28.449527 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449522    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449524    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449527    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449536    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449539    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449542    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449546    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449550    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449553    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449556    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449558    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449561    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449564    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449567    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449569    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449572    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449574    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449577    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449579    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:28.450035 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449582    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449584    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449587    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449590    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449592    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449594    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449597    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449599    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449602    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449604    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449607    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449609    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449611    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449614    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449623    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449627    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449631    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449634    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449637    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:28.450492 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449641    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449644    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449648    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449651    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449653    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449656    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449659    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449662    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449665    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449667    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449670    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449672    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449675    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449677    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449680    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449682    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449685    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449687    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449690    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449692    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:28.451008 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449695    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449697    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449700    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449702    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449704    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449707    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449710    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449714    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449717    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449719    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449722    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449725    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449727    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449730    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449732    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449734    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449737    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449739    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449742    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:28.451490 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.449744    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450414    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450422    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450425    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450428    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450431    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450435    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450438    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450441    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450443    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450446    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450449    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450451    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450454    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450457    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450460    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450464    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450467    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450470    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:28.451962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450473    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450476    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450480    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450483    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450486    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450489    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450492    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450495    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450498    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450501    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450503    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450506    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450509    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450512    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450514    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450517    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450519    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450522    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450525    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450527    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450530    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:28.452438 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450533    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450535    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450538    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450542    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450544    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450547    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450549    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450552    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450554    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450557    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450560    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450562    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450565    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450568    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450570    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450573    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450575    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450579    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450582    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:28.452978 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450585    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450588    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450590    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450592    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450595    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450598    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450600    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450603    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450605    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450607    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450610    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450612    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450614    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450617    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450620    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450622    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450625    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450627    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450629    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450632    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:28.453457 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450634    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450637    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450640    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450643    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450645    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450648    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450651    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450654    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450657    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450659    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450663    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450665    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450668    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450670    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450673    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450676    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450678    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450681    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450683    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450686    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:28.453962 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.450688    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451034    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451039    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451042    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451044    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451048    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451051    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451053    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451056    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451059    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451063    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451067    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451070    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451073    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451077    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451079    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451083    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451085    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451088    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:28.454461 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451091    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451093    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451096    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451099    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451101    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451104    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451106    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451109    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451112    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451114    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451117    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451119    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451122    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451124    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451127    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451129    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451131    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451135    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451137    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451140    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451143    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:28.454943 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451145    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451148    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451150    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451153    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451158    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451160    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451164    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451168    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451170    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451173    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451176    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451179    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451181    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451184    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451186    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451189    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451191    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451195    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:28.455463 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451199    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451202    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451205    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451208    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451211    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451214    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451217    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451220    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451222    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451225    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451227    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451230    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451232    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451236    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451238    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451241    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451243    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451246    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451249    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451252    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:28.455939 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451255    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451257    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451260    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451262    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451267    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451270    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451272    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451275    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451277    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451280    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451282    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451285    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451288    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451290    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451293    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451295    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451298    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451300    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451303    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451305    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:28.456433 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451308    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.451310    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451387    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451432    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451438    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451442    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451446    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451449    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451453    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451457    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451461    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451464    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451468    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451471    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451474    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451477    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451482    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451485    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451488    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451491    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451495    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451498    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451501    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451504    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 23:36:28.456961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451508    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451511    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451514    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451517    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451520    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451523    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451526    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451529    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451532    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451535    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451539    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451542    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451545    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451548    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451551    2562 flags.go:64] FLAG: --enable-server="true"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451554    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451558    2562 flags.go:64] FLAG: --event-burst="100"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451561    2562 flags.go:64] FLAG: --event-qps="50"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451564    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451568    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451571    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451574    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451577    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451580    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451583    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 10 23:36:28.457605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451586    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451589    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451592    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451595    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451597    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451600    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451603    2562 flags.go:64] FLAG: --feature-gates=""
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451607    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451609    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451612    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451615    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451618    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451621    2562 flags.go:64] FLAG: --help="false"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451624    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451627    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451630    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451633    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451636    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451639    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451642    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451645    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451648    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451650    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 23:36:28.458228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451654    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451658    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451661    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451664    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451667    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451670    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451673    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451675    2562 flags.go:64] FLAG: --lock-file=""
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451679    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451682    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451685    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451690    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451693    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451696    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451699    2562 flags.go:64] FLAG: --logging-format="text"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451701    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451705    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451708    2562 flags.go:64] FLAG: --manifest-url=""
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451711    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451714    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451717    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451721    2562 flags.go:64] FLAG: --max-pods="110"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451724    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451727    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451730    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 23:36:28.458807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451733    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451736    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451739    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451742    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451749    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451752    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451755    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451758    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451761    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451766    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451769    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451773    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451775    2562 flags.go:64] FLAG: --port="10250"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451778    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451781    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02932829094a0b0ba"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451785    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451787    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451790    2562 flags.go:64] FLAG: --register-node="true"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451793    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451797    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451800    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451802    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451805    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451808    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451812    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 23:36:28.459472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451814    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451817    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451820    2562 flags.go:64] FLAG: --runonce="false"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451823    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451826    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451829    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451832    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451835    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451838    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451841    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451843    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451846    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451849    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451852    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451855    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451858    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451861    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451864    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451869    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451872    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451875    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451878    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451881    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451884    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451887    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 23:36:28.460100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451889    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451892    2562 flags.go:64] FLAG: --v="2"
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451896    2562 flags.go:64] FLAG: --version="false"
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451900    2562 flags.go:64] FLAG: --vmodule=""
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451903    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.451906    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452021    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452026    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452028    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452033    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452036    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452039    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452042    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452044    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452047    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452050    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452052    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452055    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452058    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452062    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452065    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:28.460717 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452067    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452070    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452073    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452076    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452078    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452081    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452084    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452086    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452089    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452092    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452094    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452097    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452099    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452102    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452104    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452107    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452110    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452113    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452115    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452118    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:28.461262 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452122    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452124    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452127    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452129    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452132    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452135    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452137    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452140    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452142    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452145    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452149    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452151    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452154    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452156    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452159    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452161    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452164    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452167    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452169    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452172    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:28.461749 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452175    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452177    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452180    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452183    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452185    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452188    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452190    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452193    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452195    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452198    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452200    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452203    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452207    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452210    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452213    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452215    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452218    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452220    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452223    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:28.462242 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452227    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452231    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452234    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452239    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452242    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452244    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452248    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452252    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452255    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452258    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452262    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452265    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452268    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452270    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452273    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452276    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452278    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452281    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452283    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:28.462703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452286    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452288    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452291    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452294    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452297    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.452299    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.452305    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.458646    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.458660    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.459351    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.459358    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459412    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459418    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459421    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459424    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:28.463216 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459427    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459430    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459434    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459436    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459440    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459442    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459446    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459448    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459454    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459457    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459460    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459462    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459466    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459469    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459472    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459475    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459478    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459482    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459486    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:28.463598 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459489    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459492    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459495    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459497    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459500    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459503    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459507    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459509    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459512    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459515    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459518    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459521    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459523    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459526    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459528    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459531    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459533    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459536    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459539    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459541    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:28.464178 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459543    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459546    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459549    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459552    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459555    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459558    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459561    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459563    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459566    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459570    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459573    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459576    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459579    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459582    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459585    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459588    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459590    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459592    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459595    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459598    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:28.464675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459601    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459604    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459607    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459609    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459612    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459614    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459617    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459620    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459622    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459625    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459627    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459630    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459632    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459635    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459637    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459640    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459643    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459645    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459648    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459650    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:28.465248 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459653    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459655    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459658    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459660    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459663    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459666    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459668    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459671    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459673    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459676    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459679    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459681    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459684    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459688    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459691    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:28.465769 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459694    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.459699    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459822    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459826    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459829    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459832    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459835    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459837    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459840    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459842    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459845    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459848    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459850    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459854    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459856    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:28.466338 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459859    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459861    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459864    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459866    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459869    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459872    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459874    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459877    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459879    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459882    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459884    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459887    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459890    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459892    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459895    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459899    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459903    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459906    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459909    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459912    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:28.466784 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459915    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459918    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459920    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459935    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459938    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459941    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459943    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459946    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459949    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459951    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459954    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459957    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459960    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459962    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459965    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459967    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459970    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459972    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459975    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459977    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:28.467286 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459980    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459982    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459985    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459987    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459990    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459992    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459995    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.459998    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460000    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460003    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460006    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460008    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460010    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460013    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460015    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460018    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460020    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460023    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460026    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460028    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:28.467772 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460031    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460033    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460036    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460038    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460041    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460044    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460046    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460049    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460052    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460054    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460057    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460059    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460062    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460064    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460067    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460070    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460072    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460076    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:28.468418 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460080    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460084    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460088    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460091    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460094    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460097    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460099    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:28.460102    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.460107    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.460196    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.461998    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.462886    2562 server.go:1019] "Starting client certificate rotation"
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.462986    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:36:28.468854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.463431    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:36:28.469256 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.466447    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:36:28.471836 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.471815    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:36:28.482011 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.481991    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 10 23:36:28.484565 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.484551    2562 log.go:25] "Validated CRI v1 image API"
Sep 10 23:36:28.487702 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.487687    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 23:36:28.490387 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.490366    2562 fs.go:135] Filesystem UUIDs: map[7920c581-46c0-443b-a7b5-441e92969ffb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 813650a4-96df-49c3-be32-9dd5f52ace7e:/dev/nvme0n1p3]
Sep 10 23:36:28.490455 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.490386    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 23:36:28.493697 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.493678    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:36:28.495726 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.495624    2562 manager.go:216] Machine: {Timestamp:2026-09-10 23:36:28.494301701 +0000 UTC m=+0.325329247 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099962 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27c4decea0aa157264f42a5bddb1a6 SystemUUID:ec27c4de-cea0-aa15-7264-f42a5bddb1a6 BootID:1ef75930-c0d6-44fe-9b5c-cb26da509f9d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:d3:62:a6:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:d3:62:a6:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:56:87:e1:4a:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 23:36:28.495726 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.495724    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 23:36:28.495834 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.495796    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 23:36:28.498556 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.498537    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 23:36:28.498677 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.498558    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-218.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 23:36:28.499260 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.499250    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 23:36:28.499290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.499261    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 23:36:28.499290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.499269    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:36:28.499873 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.499864    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:36:28.500525 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.500516    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 23:36:28.502978 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.502968    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:36:28.503081 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.503072    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 23:36:28.503121 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.503114    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 23:36:28.503152 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.503125    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 23:36:28.503152 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.503136    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 23:36:28.503152 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.503146    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 10 23:36:28.503229 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.503154    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 23:36:28.503572 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.503557    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:36:28.503630 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.503587    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:36:28.506284 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.506265    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 23:36:28.509463 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.509446    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 23:36:28.509537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.509477    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 23:36:28.510891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510863    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 23:36:28.510891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510895    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510906    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510912    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510938    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510947    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510964    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510969    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510977    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510983    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510991    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 23:36:28.511032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.510996    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 23:36:28.511673 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.511664    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 23:36:28.511673 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.511673    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 23:36:28.512993 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.512958    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 23:36:28.513077 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.512993    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 23:36:28.513118 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.513087    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-218.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 23:36:28.515370 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.515358    2562 server.go:1268] "Started kubelet"
Sep 10 23:36:28.515442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.515396    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 23:36:28.515507 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.515440    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 23:36:28.515559 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.515527    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 23:36:28.516268 ip-10-0-137-218 systemd[1]: Started Kubernetes Kubelet.
Sep 10 23:36:28.516693 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.516677    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 23:36:28.517209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.517196    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.521065    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.521574    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.521670    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522026    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522037    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522057    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522225    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522235    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.522325    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522476    2562 factory.go:55] Registering systemd factory
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522488    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522664    2562 factory.go:153] Registering CRI-O factory
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522678    2562 factory.go:223] Registration of the crio container factory successfully
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522733    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522747    2562 factory.go:103] Registering Raw factory
Sep 10 23:36:28.522865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.522758    2562 manager.go:1196] Started watching for new ooms in manager
Sep 10 23:36:28.523383 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.523268    2562 manager.go:349] Starting recovery of all containers
Sep 10 23:36:28.530318 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.530292    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 23:36:28.530412 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.530321    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 23:36:28.535020 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.530361    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-218.ec2.internal.18d419dce6023538  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-218.ec2.internal,UID:ip-10-0-137-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-218.ec2.internal,},FirstTimestamp:2026-09-10 23:36:28.515333432 +0000 UTC m=+0.346360956,LastTimestamp:2026-09-10 23:36:28.515333432 +0000 UTC m=+0.346360956,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-218.ec2.internal,}"
Sep 10 23:36:28.539295 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.539277    2562 manager.go:354] Recovery completed
Sep 10 23:36:28.541458 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.541438    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-95ndh"
Sep 10 23:36:28.544478 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.544467    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:28.546964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.546283    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:28.546964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.546310    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:28.546964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.546320    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:28.546964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.546801    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 23:36:28.546964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.546812    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 23:36:28.546964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.546827    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:36:28.546964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.546813    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:28.547551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.547535    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:28.547615 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.547562    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:28.547615 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.547573    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:28.548059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.548044    2562 policy_none.go:49] "None policy: Start"
Sep 10 23:36:28.548133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.548062    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 23:36:28.548133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.548072    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 23:36:28.548351 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.548338    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-95ndh"
Sep 10 23:36:28.549442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.549431    2562 policy_none.go:47] "Start"
Sep 10 23:36:28.590568 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.590553    2562 manager.go:337] "Starting Device Plugin manager"
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.590583    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.590593    2562 server.go:87] "Starting device plugin registration server"
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.590797    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.590809    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.590907    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.591036    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.591047    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.591468    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 23:36:28.601148 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.591509    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:28.647783 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.647753    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 23:36:28.648879 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.648865    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 23:36:28.648966 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.648913    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 23:36:28.648966 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.648961    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 23:36:28.649057 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.649037    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 23:36:28.650879 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.650861    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:28.691587 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.691570    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:28.692286 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.692271    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:28.692365 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.692298    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:28.692365 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.692308    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:28.692365 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.692327    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.697810 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.697773    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.697810 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.697790    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-218.ec2.internal\": node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:28.718699 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.718683    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:28.749610 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.749589    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal"]
Sep 10 23:36:28.749677 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.749646    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:28.750374 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.750361    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:28.750442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.750384    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:28.750442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.750398    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:28.752629 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.752618    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:28.752742 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.752730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.752779 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.752774    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:28.753302 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.753284    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:28.753366 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.753314    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:28.753366 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.753332    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:28.753438 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.753295    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:28.753438 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.753395    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:28.753438 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.753406    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:28.756447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.756429    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.756524 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.756453    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:28.757145 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.757127    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:28.757231 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.757152    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:28.757231 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.757164    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:28.782565 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.782437    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-218.ec2.internal\" not found" node="ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.786586 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.786571    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-218.ec2.internal\" not found" node="ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.819272 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.819255    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:28.824579 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.824566    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e5c14c36611ccf2603ecc0ec2faf0be-config\") pod \"kube-apiserver-proxy-ip-10-0-137-218.ec2.internal\" (UID: \"4e5c14c36611ccf2603ecc0ec2faf0be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.824631 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.824586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb2f78d5c62bf4a2e472052e97851392-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal\" (UID: \"bb2f78d5c62bf4a2e472052e97851392\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.825018 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.825004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2f78d5c62bf4a2e472052e97851392-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal\" (UID: \"bb2f78d5c62bf4a2e472052e97851392\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.919725 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:28.919698    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:28.926035 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.926015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e5c14c36611ccf2603ecc0ec2faf0be-config\") pod \"kube-apiserver-proxy-ip-10-0-137-218.ec2.internal\" (UID: \"4e5c14c36611ccf2603ecc0ec2faf0be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.926102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.926043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb2f78d5c62bf4a2e472052e97851392-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal\" (UID: \"bb2f78d5c62bf4a2e472052e97851392\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.926102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.926068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2f78d5c62bf4a2e472052e97851392-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal\" (UID: \"bb2f78d5c62bf4a2e472052e97851392\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.926102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.926097    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e5c14c36611ccf2603ecc0ec2faf0be-config\") pod \"kube-apiserver-proxy-ip-10-0-137-218.ec2.internal\" (UID: \"4e5c14c36611ccf2603ecc0ec2faf0be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.926218 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.926103    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2f78d5c62bf4a2e472052e97851392-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal\" (UID: \"bb2f78d5c62bf4a2e472052e97851392\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:28.926218 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:28.926112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb2f78d5c62bf4a2e472052e97851392-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal\" (UID: \"bb2f78d5c62bf4a2e472052e97851392\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:29.020446 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:29.020407    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:29.084836 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.084813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:29.089627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.089428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:29.120805 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:29.120781    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:29.221213 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:29.221193    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:29.322132 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:29.322110    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:29.422259 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:29.422239    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:29.447184 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.447158    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:29.462490 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.462467    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 23:36:29.462611 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.462580    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:36:29.462651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.462613    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:36:29.523813 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:29.523785    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:29.552090 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.552040    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 23:31:29 +0000 UTC" deadline="2028-02-10 08:29:49.375276718 +0000 UTC"
Sep 10 23:36:29.552090 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.552069    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12416h53m19.823211164s"
Sep 10 23:36:29.624854 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:29.624829    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-218.ec2.internal\" not found"
Sep 10 23:36:29.697879 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.697853    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:29.709755 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:29.709230    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5c14c36611ccf2603ecc0ec2faf0be.slice/crio-95af22ef093790d180d4d08597de780e5d25c92f43d8891479eca1674655c3bc WatchSource:0}: Error finding container 95af22ef093790d180d4d08597de780e5d25c92f43d8891479eca1674655c3bc: Status 404 returned error can't find the container with id 95af22ef093790d180d4d08597de780e5d25c92f43d8891479eca1674655c3bc
Sep 10 23:36:29.711656 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:29.711627    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2f78d5c62bf4a2e472052e97851392.slice/crio-8c733abb97950724ca0c75796c1c0537d9a6f601e7af9d02a3e81a2fa55ae8df WatchSource:0}: Error finding container 8c733abb97950724ca0c75796c1c0537d9a6f601e7af9d02a3e81a2fa55ae8df: Status 404 returned error can't find the container with id 8c733abb97950724ca0c75796c1c0537d9a6f601e7af9d02a3e81a2fa55ae8df
Sep 10 23:36:29.715925 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.715910    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:36:29.722057 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.722038    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:29.734154 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.734137    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:36:29.736274 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.736261    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal"
Sep 10 23:36:29.744356 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.744338    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:36:29.751181 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:29.751160    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:30.504974 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.504919    2562 apiserver.go:52] "Watching apiserver"
Sep 10 23:36:30.515622 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.515588    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 23:36:30.516942 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.516911    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-gvg49","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal","openshift-network-operator/iptables-alerter-zj52b","kube-system/konnectivity-agent-7br8t","openshift-cluster-node-tuning-operator/tuned-zsf6w","openshift-image-registry/node-ca-2q2tj","openshift-multus/multus-2lmmn","openshift-multus/multus-additional-cni-plugins-ll288","openshift-multus/network-metrics-daemon-m8gj9","openshift-network-diagnostics/network-check-target-27vsj","openshift-ovn-kubernetes/ovnkube-node-x8sj7","kube-system/global-pull-secret-syncer-qmbgq","kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"]
Sep 10 23:36:30.519961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.519940    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.523068 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.522377    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 23:36:30.523068 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.522999    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.524177 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.523728    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 23:36:30.524177 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.523829    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 23:36:30.524177 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.524060    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cj4dq\""
Sep 10 23:36:30.525463 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.525440    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 23:36:30.525683 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.525666    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 23:36:30.525820 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.525801    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t5ldc\""
Sep 10 23:36:30.525916 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.525901    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 23:36:30.528607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.526044    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 23:36:30.528607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.526231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.528607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.526454    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 23:36:30.528607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.526234    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 23:36:30.528607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.526680    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:30.528607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.526768    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 23:36:30.528607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.528511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:30.529461 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.529253    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 23:36:30.531778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.530266    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:36:30.531778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.530557    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6n45x\""
Sep 10 23:36:30.531778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.530670    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 23:36:30.531778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.530562    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 23:36:30.531778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.530991    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 23:36:30.531778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.531096    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2hpfs\""
Sep 10 23:36:30.531778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.531282    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.531778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.531312    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-cni-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhcf\" (UniqueName: \"kubernetes.io/projected/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-kube-api-access-mwhcf\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-systemd-units\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533549    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-slash\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-ovn\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdxl\" (UniqueName: \"kubernetes.io/projected/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-kube-api-access-8mdxl\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-daemon-config\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-run-netns\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovnkube-config\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-k8s-cni-cncf-io\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-cni-bin\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-hostroot\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-multus-certs\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533796    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-node-log\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.534898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-cni-bin\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovn-node-metrics-cert\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-kubelet\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-cni-netd\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.533990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-system-cni-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-cnibin\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534069    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2cf86d6f-6f67-40a4-98bf-f97c338816b3-cni-binary-copy\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-socket-dir-parent\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-cni-multus\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-kubelet\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-var-lib-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovnkube-script-lib\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-conf-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-systemd\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-env-overrides\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.535788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-iptables-alerter-script\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.536612 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-host-slash\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.536612 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-log-socket\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.536612 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-netns\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.536612 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-etc-kubernetes\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.536612 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534496    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h99tm\" (UniqueName: \"kubernetes.io/projected/2cf86d6f-6f67-40a4-98bf-f97c338816b3-kube-api-access-h99tm\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.536612 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.534519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-etc-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.539048 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.538184    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hb9kx\""
Sep 10 23:36:30.539048 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.538462    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 23:36:30.539048 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.538647    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:36:30.539048 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.538882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.539780 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.539764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:30.539871 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.539847    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:30.541827 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.541807    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:36:30.542915 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.542896    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 23:36:30.550170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.546474    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.550170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.549093    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 23:36:30.550170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.549466    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.550170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.549519    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vkhdv\""
Sep 10 23:36:30.552417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.551598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wdj8v\""
Sep 10 23:36:30.552417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.551691    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 23:36:30.552417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.551866    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 23:36:30.552417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.551923    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 23:36:30.552853 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.552723    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 23:36:30.552853 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.552742    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:30.552853 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.552813    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:30.555270 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.553328    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 23:36:30.555270 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.554083    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x9st2\""
Sep 10 23:36:30.555270 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.553330    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 23:36:30.556793 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.556773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.559867 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.559848    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mwmdk\""
Sep 10 23:36:30.560119 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.560100    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 23:36:30.560373 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.560330    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 23:36:30.560709 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.560692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.560945 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.560926    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6s2n5"
Sep 10 23:36:30.561099 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.561076    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:30.569389 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.569308    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6s2n5"
Sep 10 23:36:30.623577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.623537    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.634969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-var-lib-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovnkube-script-lib\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-systemd\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635060    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-iptables-alerter-script\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-host-slash\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e324660-2423-4acb-becf-9a78be67198a-hosts-file\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-log-socket\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-netns\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-etc-kubernetes\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrz8\" (UniqueName: \"kubernetes.io/projected/2f1e94ed-d8d5-47da-8a95-072c5848b218-kube-api-access-fgrz8\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-cnibin\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-cni-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-etc-kubernetes\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-host-slash\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.636919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-log-socket\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-sys-fs\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-cni-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635635    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-tuned\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635637    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-systemd\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-var-lib-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-netns\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635679    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.635711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhcf\" (UniqueName: \"kubernetes.io/projected/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-kube-api-access-mwhcf\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-systemd-units\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-slash\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-daemon-config\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-systemd\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636164    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad3a3d17-1e95-41bb-89a3-4f74e903e750-tmp\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4962449-bcd3-4306-b62a-0de981ea1bdc-konnectivity-ca\") pod \"konnectivity-agent-7br8t\" (UID: \"d4962449-bcd3-4306-b62a-0de981ea1bdc\") " pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636214    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-kubernetes\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-run-netns\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.637831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-iptables-alerter-script\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovnkube-config\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-k8s-cni-cncf-io\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovnkube-script-lib\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-multus-certs\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-systemd-units\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e324660-2423-4acb-becf-9a78be67198a-tmp-dir\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-run-netns\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-host\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-k8s-cni-cncf-io\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-slash\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-run-multus-certs\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f664w\" (UniqueName: \"kubernetes.io/projected/e48f204d-76c2-43fa-8100-486c6a3e31a3-kube-api-access-f664w\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-node-log\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-cni-bin\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-socket-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636782    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.638757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-cni-netd\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-system-cni-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-socket-dir-parent\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-cni-multus\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-cni-bin\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-cni-netd\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636991    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-kubelet\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.636900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovnkube-config\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-node-log\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-sys\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-dbus\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637195    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-system-cni-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4962449-bcd3-4306-b62a-0de981ea1bdc-agent-certs\") pod \"konnectivity-agent-7br8t\" (UID: \"d4962449-bcd3-4306-b62a-0de981ea1bdc\") " pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-socket-dir-parent\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637289    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-kubelet\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-conf-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.639672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-cni-multus\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637359    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-conf-dir\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637427    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637476    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qvc\" (UniqueName: \"kubernetes.io/projected/ad3a3d17-1e95-41bb-89a3-4f74e903e750-kube-api-access-v2qvc\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637525    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2cf86d6f-6f67-40a4-98bf-f97c338816b3-multus-daemon-config\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5bx\" (UniqueName: \"kubernetes.io/projected/745744e6-2b42-4e82-bfba-6c2f79f57633-kube-api-access-mc5bx\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-env-overrides\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-device-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-etc-selinux\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysctl-conf\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-var-lib-kubelet\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-host\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637815    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvcnb\" (UniqueName: \"kubernetes.io/projected/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-kube-api-access-zvcnb\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637853    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h99tm\" (UniqueName: \"kubernetes.io/projected/2cf86d6f-6f67-40a4-98bf-f97c338816b3-kube-api-access-h99tm\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.640619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzxmc\" (UniqueName: \"kubernetes.io/projected/1e324660-2423-4acb-becf-9a78be67198a-kube-api-access-gzxmc\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysconfig\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-etc-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-env-overrides\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-etc-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.637986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-registration-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysctl-d\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638147    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-lib-modules\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638164    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-serviceca\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-ovn\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-ovn\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdxl\" (UniqueName: \"kubernetes.io/projected/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-kube-api-access-8mdxl\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-run\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-cni-bin\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-hostroot\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.641434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-hostroot\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-host-var-lib-cni-bin\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovn-node-metrics-cert\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-modprobe-d\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-run-openvswitch\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-kubelet-config\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-kubelet\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638847    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-host-kubelet\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-cnibin\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2cf86d6f-6f67-40a4-98bf-f97c338816b3-cni-binary-copy\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638945    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.638962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2cf86d6f-6f67-40a4-98bf-f97c338816b3-cnibin\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.642290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.639475    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2cf86d6f-6f67-40a4-98bf-f97c338816b3-cni-binary-copy\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.646729 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.646704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhcf\" (UniqueName: \"kubernetes.io/projected/5224f0ca-4aee-4ea1-9c5e-6c8ae526192b-kube-api-access-mwhcf\") pod \"iptables-alerter-zj52b\" (UID: \"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b\") " pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.649352 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.649327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h99tm\" (UniqueName: \"kubernetes.io/projected/2cf86d6f-6f67-40a4-98bf-f97c338816b3-kube-api-access-h99tm\") pod \"multus-2lmmn\" (UID: \"2cf86d6f-6f67-40a4-98bf-f97c338816b3\") " pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.649921 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.649894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-ovn-node-metrics-cert\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.651952 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.651916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdxl\" (UniqueName: \"kubernetes.io/projected/fc61cf8b-3a02-49fd-b1f5-92ddd99729ac-kube-api-access-8mdxl\") pod \"ovnkube-node-x8sj7\" (UID: \"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.654055 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.654008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal" event={"ID":"4e5c14c36611ccf2603ecc0ec2faf0be","Type":"ContainerStarted","Data":"95af22ef093790d180d4d08597de780e5d25c92f43d8891479eca1674655c3bc"}
Sep 10 23:36:30.655949 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.655892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal" event={"ID":"bb2f78d5c62bf4a2e472052e97851392","Type":"ContainerStarted","Data":"8c733abb97950724ca0c75796c1c0537d9a6f601e7af9d02a3e81a2fa55ae8df"}
Sep 10 23:36:30.739891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.739857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-systemd\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.739901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad3a3d17-1e95-41bb-89a3-4f74e903e750-tmp\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.739925    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4962449-bcd3-4306-b62a-0de981ea1bdc-konnectivity-ca\") pod \"konnectivity-agent-7br8t\" (UID: \"d4962449-bcd3-4306-b62a-0de981ea1bdc\") " pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:30.740092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.739959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-kubernetes\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.739985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e324660-2423-4acb-becf-9a78be67198a-tmp-dir\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.740092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-host\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f664w\" (UniqueName: \"kubernetes.io/projected/e48f204d-76c2-43fa-8100-486c6a3e31a3-kube-api-access-f664w\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.740092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-socket-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.740092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-sys\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-dbus\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4962449-bcd3-4306-b62a-0de981ea1bdc-agent-certs\") pod \"konnectivity-agent-7br8t\" (UID: \"d4962449-bcd3-4306-b62a-0de981ea1bdc\") " pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qvc\" (UniqueName: \"kubernetes.io/projected/ad3a3d17-1e95-41bb-89a3-4f74e903e750-kube-api-access-v2qvc\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5bx\" (UniqueName: \"kubernetes.io/projected/745744e6-2b42-4e82-bfba-6c2f79f57633-kube-api-access-mc5bx\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-device-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740308    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-etc-selinux\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysctl-conf\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-var-lib-kubelet\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-var-lib-kubelet\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-dbus\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.740509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740501    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-host\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740525    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-socket-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-device-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740667    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e324660-2423-4acb-becf-9a78be67198a-tmp-dir\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-etc-selinux\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-host\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-kubernetes\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.740795    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740878    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-host\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvcnb\" (UniqueName: \"kubernetes.io/projected/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-kube-api-access-zvcnb\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-sys\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysctl-conf\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.740883    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret podName:3c15c11b-b1b2-4c3b-a4c8-edc70777af3d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:31.240868027 +0000 UTC m=+3.071895538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret") pod "global-pull-secret-syncer-qmbgq" (UID: "3c15c11b-b1b2-4c3b-a4c8-edc70777af3d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxmc\" (UniqueName: \"kubernetes.io/projected/1e324660-2423-4acb-becf-9a78be67198a-kube-api-access-gzxmc\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.740999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysconfig\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-registration-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.741298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741029    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4962449-bcd3-4306-b62a-0de981ea1bdc-konnectivity-ca\") pod \"konnectivity-agent-7br8t\" (UID: \"d4962449-bcd3-4306-b62a-0de981ea1bdc\") " pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysctl-d\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-lib-modules\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysconfig\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-sysctl-d\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-lib-modules\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-registration-dir\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-serviceca\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-run\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-modprobe-d\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-kubelet-config\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e324660-2423-4acb-becf-9a78be67198a-hosts-file\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrz8\" (UniqueName: \"kubernetes.io/projected/2f1e94ed-d8d5-47da-8a95-072c5848b218-kube-api-access-fgrz8\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.742196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-cnibin\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-serviceca\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-sys-fs\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-tuned\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-kubelet-config\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-system-cni-dir\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e324660-2423-4acb-becf-9a78be67198a-hosts-file\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.741914    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.741951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f1e94ed-d8d5-47da-8a95-072c5848b218-sys-fs\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.742044    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs podName:745744e6-2b42-4e82-bfba-6c2f79f57633 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:31.242026301 +0000 UTC m=+3.073053830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs") pod "network-metrics-daemon-m8gj9" (UID: "745744e6-2b42-4e82-bfba-6c2f79f57633") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.742250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.742309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-run\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.742340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e48f204d-76c2-43fa-8100-486c6a3e31a3-cnibin\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.743074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.742464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-modprobe-d\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.744023 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.742504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e48f204d-76c2-43fa-8100-486c6a3e31a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.744023 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.742579    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-systemd\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.745415 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.744919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad3a3d17-1e95-41bb-89a3-4f74e903e750-etc-tuned\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.745830 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.745790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad3a3d17-1e95-41bb-89a3-4f74e903e750-tmp\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.747119 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.747096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4962449-bcd3-4306-b62a-0de981ea1bdc-agent-certs\") pod \"konnectivity-agent-7br8t\" (UID: \"d4962449-bcd3-4306-b62a-0de981ea1bdc\") " pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:30.748031 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.747919    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:30.748031 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.747952    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:30.748031 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.747970    2562 projected.go:196] Error preparing data for projected volume kube-api-access-x94c8 for pod openshift-network-diagnostics/network-check-target-27vsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:30.748246 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:30.748105    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8 podName:333a5c94-32b1-4b76-9e90-39206904ddce nodeName:}" failed. No retries permitted until 2026-09-10 23:36:31.248086731 +0000 UTC m=+3.079114268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x94c8" (UniqueName: "kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8") pod "network-check-target-27vsj" (UID: "333a5c94-32b1-4b76-9e90-39206904ddce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:30.754456 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.754276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrz8\" (UniqueName: \"kubernetes.io/projected/2f1e94ed-d8d5-47da-8a95-072c5848b218-kube-api-access-fgrz8\") pod \"aws-ebs-csi-driver-node-j92r5\" (UID: \"2f1e94ed-d8d5-47da-8a95-072c5848b218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.754456 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.754337    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxmc\" (UniqueName: \"kubernetes.io/projected/1e324660-2423-4acb-becf-9a78be67198a-kube-api-access-gzxmc\") pod \"node-resolver-gvg49\" (UID: \"1e324660-2423-4acb-becf-9a78be67198a\") " pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:30.754456 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.754411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f664w\" (UniqueName: \"kubernetes.io/projected/e48f204d-76c2-43fa-8100-486c6a3e31a3-kube-api-access-f664w\") pod \"multus-additional-cni-plugins-ll288\" (UID: \"e48f204d-76c2-43fa-8100-486c6a3e31a3\") " pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.756203 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.756150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qvc\" (UniqueName: \"kubernetes.io/projected/ad3a3d17-1e95-41bb-89a3-4f74e903e750-kube-api-access-v2qvc\") pod \"tuned-zsf6w\" (UID: \"ad3a3d17-1e95-41bb-89a3-4f74e903e750\") " pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.757302 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.756674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvcnb\" (UniqueName: \"kubernetes.io/projected/6f736cd6-ad74-4550-a2a8-82f59ee9d98f-kube-api-access-zvcnb\") pod \"node-ca-2q2tj\" (UID: \"6f736cd6-ad74-4550-a2a8-82f59ee9d98f\") " pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.758941 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.758909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5bx\" (UniqueName: \"kubernetes.io/projected/745744e6-2b42-4e82-bfba-6c2f79f57633-kube-api-access-mc5bx\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:30.849200 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.849170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2lmmn"
Sep 10 23:36:30.860146 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.860126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:30.872048 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.872019    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zj52b"
Sep 10 23:36:30.880937 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.880914    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:30.891792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.891769    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsf6w"
Sep 10 23:36:30.900695 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.900671    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2q2tj"
Sep 10 23:36:30.907767 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.907744    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5"
Sep 10 23:36:30.917574 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.917554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ll288"
Sep 10 23:36:30.926570 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:30.926548    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gvg49"
Sep 10 23:36:31.245248 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:31.245209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:31.245419 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:31.245365    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:31.245505 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:31.245363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:31.245505 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:31.245440    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs podName:745744e6-2b42-4e82-bfba-6c2f79f57633 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:32.245419454 +0000 UTC m=+4.076446988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs") pod "network-metrics-daemon-m8gj9" (UID: "745744e6-2b42-4e82-bfba-6c2f79f57633") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:31.245505 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:31.245465    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:31.245666 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:31.245517    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret podName:3c15c11b-b1b2-4c3b-a4c8-edc70777af3d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:32.245501031 +0000 UTC m=+4.076528548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret") pod "global-pull-secret-syncer-qmbgq" (UID: "3c15c11b-b1b2-4c3b-a4c8-edc70777af3d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:31.253932 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:31.253910    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:31.346484 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:31.346454    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:31.346639 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:31.346619    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:31.346691 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:31.346645    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:31.346691 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:31.346660    2562 projected.go:196] Error preparing data for projected volume kube-api-access-x94c8 for pod openshift-network-diagnostics/network-check-target-27vsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:31.346790 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:31.346714    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8 podName:333a5c94-32b1-4b76-9e90-39206904ddce nodeName:}" failed. No retries permitted until 2026-09-10 23:36:32.346696442 +0000 UTC m=+4.177723966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x94c8" (UniqueName: "kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8") pod "network-check-target-27vsj" (UID: "333a5c94-32b1-4b76-9e90-39206904ddce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:31.572642 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:31.572549    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:31 +0000 UTC" deadline="2028-04-04 06:31:23.320730713 +0000 UTC"
Sep 10 23:36:31.572642 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:31.572587    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13710h54m51.748150663s"
Sep 10 23:36:31.831882 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.831851    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc61cf8b_3a02_49fd_b1f5_92ddd99729ac.slice/crio-ce53a78f23ba2443a5f96bec5b10e6a6623b347791c63629d0cdbafecba2fe2b WatchSource:0}: Error finding container ce53a78f23ba2443a5f96bec5b10e6a6623b347791c63629d0cdbafecba2fe2b: Status 404 returned error can't find the container with id ce53a78f23ba2443a5f96bec5b10e6a6623b347791c63629d0cdbafecba2fe2b
Sep 10 23:36:31.834443 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.834329    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5224f0ca_4aee_4ea1_9c5e_6c8ae526192b.slice/crio-71e9b56275405b0fa439db3ae58cb0b3aba1ded5a4c7026f01336191de2163d0 WatchSource:0}: Error finding container 71e9b56275405b0fa439db3ae58cb0b3aba1ded5a4c7026f01336191de2163d0: Status 404 returned error can't find the container with id 71e9b56275405b0fa439db3ae58cb0b3aba1ded5a4c7026f01336191de2163d0
Sep 10 23:36:31.857675 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.857652    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e324660_2423_4acb_becf_9a78be67198a.slice/crio-31fa70934e7592503e65a6e447710bc88ac704e0f83b1b798585a644df1a676a WatchSource:0}: Error finding container 31fa70934e7592503e65a6e447710bc88ac704e0f83b1b798585a644df1a676a: Status 404 returned error can't find the container with id 31fa70934e7592503e65a6e447710bc88ac704e0f83b1b798585a644df1a676a
Sep 10 23:36:31.859318 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.859290    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf86d6f_6f67_40a4_98bf_f97c338816b3.slice/crio-08c18db0284252c4e606363b626b35f849ba3c489572b37179d65d1e00b9fc50 WatchSource:0}: Error finding container 08c18db0284252c4e606363b626b35f849ba3c489572b37179d65d1e00b9fc50: Status 404 returned error can't find the container with id 08c18db0284252c4e606363b626b35f849ba3c489572b37179d65d1e00b9fc50
Sep 10 23:36:31.860287 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.860261    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48f204d_76c2_43fa_8100_486c6a3e31a3.slice/crio-bc659294e0f5698273c7264825a2b9e6b0a75cf788e5209f2b8d25a92e6d2016 WatchSource:0}: Error finding container bc659294e0f5698273c7264825a2b9e6b0a75cf788e5209f2b8d25a92e6d2016: Status 404 returned error can't find the container with id bc659294e0f5698273c7264825a2b9e6b0a75cf788e5209f2b8d25a92e6d2016
Sep 10 23:36:31.862350 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.862075    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3a3d17_1e95_41bb_89a3_4f74e903e750.slice/crio-c1bd0716ac0317d5594d70eb859cd32f85780ac95ea31a5d6f29509bc21bd114 WatchSource:0}: Error finding container c1bd0716ac0317d5594d70eb859cd32f85780ac95ea31a5d6f29509bc21bd114: Status 404 returned error can't find the container with id c1bd0716ac0317d5594d70eb859cd32f85780ac95ea31a5d6f29509bc21bd114
Sep 10 23:36:31.862831 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.862809    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f736cd6_ad74_4550_a2a8_82f59ee9d98f.slice/crio-cc5b8c1f16d484b8b70f39cf9f462db27bf728857c7fe27bbf287ef825ceba26 WatchSource:0}: Error finding container cc5b8c1f16d484b8b70f39cf9f462db27bf728857c7fe27bbf287ef825ceba26: Status 404 returned error can't find the container with id cc5b8c1f16d484b8b70f39cf9f462db27bf728857c7fe27bbf287ef825ceba26
Sep 10 23:36:31.863803 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.863732    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1e94ed_d8d5_47da_8a95_072c5848b218.slice/crio-3034812a27e2d564baf43d680b6d001420534fcaf3bfdc9a1c85ae8e55987d74 WatchSource:0}: Error finding container 3034812a27e2d564baf43d680b6d001420534fcaf3bfdc9a1c85ae8e55987d74: Status 404 returned error can't find the container with id 3034812a27e2d564baf43d680b6d001420534fcaf3bfdc9a1c85ae8e55987d74
Sep 10 23:36:31.866302 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:36:31.865699    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4962449_bcd3_4306_b62a_0de981ea1bdc.slice/crio-34c0e9ab5f017abbfea56482949d65aa1197f4b34978ce22161bbe3fb37da6a6 WatchSource:0}: Error finding container 34c0e9ab5f017abbfea56482949d65aa1197f4b34978ce22161bbe3fb37da6a6: Status 404 returned error can't find the container with id 34c0e9ab5f017abbfea56482949d65aa1197f4b34978ce22161bbe3fb37da6a6
Sep 10 23:36:32.252717 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.252475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:32.252866 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.252728    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:32.252866 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.252760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:32.252866 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.252789    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs podName:745744e6-2b42-4e82-bfba-6c2f79f57633 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:34.252770954 +0000 UTC m=+6.083798475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs") pod "network-metrics-daemon-m8gj9" (UID: "745744e6-2b42-4e82-bfba-6c2f79f57633") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:32.252866 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.252844    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:32.253094 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.252882    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret podName:3c15c11b-b1b2-4c3b-a4c8-edc70777af3d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:34.252869251 +0000 UTC m=+6.083896777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret") pod "global-pull-secret-syncer-qmbgq" (UID: "3c15c11b-b1b2-4c3b-a4c8-edc70777af3d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:32.353718 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.353631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:32.353894 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.353874    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:32.353965 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.353901    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:32.353965 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.353915    2562 projected.go:196] Error preparing data for projected volume kube-api-access-x94c8 for pod openshift-network-diagnostics/network-check-target-27vsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:32.354100 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.353990    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8 podName:333a5c94-32b1-4b76-9e90-39206904ddce nodeName:}" failed. No retries permitted until 2026-09-10 23:36:34.353954733 +0000 UTC m=+6.184982253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x94c8" (UniqueName: "kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8") pod "network-check-target-27vsj" (UID: "333a5c94-32b1-4b76-9e90-39206904ddce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:32.573542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.573472    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:31 +0000 UTC" deadline="2028-03-10 17:56:06.946813174 +0000 UTC"
Sep 10 23:36:32.573542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.573538    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13122h19m34.373279013s"
Sep 10 23:36:32.650200 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.650124    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:32.650353 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.650252    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:32.650353 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.650286    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:32.650469 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.650405    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:32.650469 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.650135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:32.650568 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:32.650503    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:32.680816 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.680778    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2q2tj" event={"ID":"6f736cd6-ad74-4550-a2a8-82f59ee9d98f","Type":"ContainerStarted","Data":"cc5b8c1f16d484b8b70f39cf9f462db27bf728857c7fe27bbf287ef825ceba26"}
Sep 10 23:36:32.684197 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.684171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsf6w" event={"ID":"ad3a3d17-1e95-41bb-89a3-4f74e903e750","Type":"ContainerStarted","Data":"c1bd0716ac0317d5594d70eb859cd32f85780ac95ea31a5d6f29509bc21bd114"}
Sep 10 23:36:32.700295 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.700268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zj52b" event={"ID":"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b","Type":"ContainerStarted","Data":"71e9b56275405b0fa439db3ae58cb0b3aba1ded5a4c7026f01336191de2163d0"}
Sep 10 23:36:32.704726 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.704619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"ce53a78f23ba2443a5f96bec5b10e6a6623b347791c63629d0cdbafecba2fe2b"}
Sep 10 23:36:32.710363 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.710334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal" event={"ID":"4e5c14c36611ccf2603ecc0ec2faf0be","Type":"ContainerStarted","Data":"557729d442b200883493810511eb33021baba56c3750209265a63f9e27350604"}
Sep 10 23:36:32.729653 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.729625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7br8t" event={"ID":"d4962449-bcd3-4306-b62a-0de981ea1bdc","Type":"ContainerStarted","Data":"34c0e9ab5f017abbfea56482949d65aa1197f4b34978ce22161bbe3fb37da6a6"}
Sep 10 23:36:32.734299 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.734273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll288" event={"ID":"e48f204d-76c2-43fa-8100-486c6a3e31a3","Type":"ContainerStarted","Data":"bc659294e0f5698273c7264825a2b9e6b0a75cf788e5209f2b8d25a92e6d2016"}
Sep 10 23:36:32.742797 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.742768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2lmmn" event={"ID":"2cf86d6f-6f67-40a4-98bf-f97c338816b3","Type":"ContainerStarted","Data":"08c18db0284252c4e606363b626b35f849ba3c489572b37179d65d1e00b9fc50"}
Sep 10 23:36:32.762646 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.762576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gvg49" event={"ID":"1e324660-2423-4acb-becf-9a78be67198a","Type":"ContainerStarted","Data":"31fa70934e7592503e65a6e447710bc88ac704e0f83b1b798585a644df1a676a"}
Sep 10 23:36:32.767686 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.767659    2562 generic.go:356] "Generic (PLEG): container finished" podID="bb2f78d5c62bf4a2e472052e97851392" containerID="049b440669293e4ec1d9d1de2107f72a6cd3eff5d397f5b512cbb8c425739763" exitCode=0
Sep 10 23:36:32.767804 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.767738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal" event={"ID":"bb2f78d5c62bf4a2e472052e97851392","Type":"ContainerDied","Data":"049b440669293e4ec1d9d1de2107f72a6cd3eff5d397f5b512cbb8c425739763"}
Sep 10 23:36:32.777624 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.777600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5" event={"ID":"2f1e94ed-d8d5-47da-8a95-072c5848b218","Type":"ContainerStarted","Data":"3034812a27e2d564baf43d680b6d001420534fcaf3bfdc9a1c85ae8e55987d74"}
Sep 10 23:36:32.789723 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:32.789101    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-218.ec2.internal" podStartSLOduration=2.789082913 podStartE2EDuration="2.789082913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:32.727809028 +0000 UTC m=+4.558836566" watchObservedRunningTime="2026-09-10 23:36:32.789082913 +0000 UTC m=+4.620110452"
Sep 10 23:36:33.792933 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:33.792849    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal" event={"ID":"bb2f78d5c62bf4a2e472052e97851392","Type":"ContainerStarted","Data":"591c262fdf088a8c9eb106e2db822a9733a8faf0e8af2d818846973dcbf7c784"}
Sep 10 23:36:34.273318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:34.273274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:34.273510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:34.273338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:34.273510 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.273505    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:34.273637 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.273572    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret podName:3c15c11b-b1b2-4c3b-a4c8-edc70777af3d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:38.273552969 +0000 UTC m=+10.104580487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret") pod "global-pull-secret-syncer-qmbgq" (UID: "3c15c11b-b1b2-4c3b-a4c8-edc70777af3d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:34.274024 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.273989    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:34.274118 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.274060    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs podName:745744e6-2b42-4e82-bfba-6c2f79f57633 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:38.274044622 +0000 UTC m=+10.105072152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs") pod "network-metrics-daemon-m8gj9" (UID: "745744e6-2b42-4e82-bfba-6c2f79f57633") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:34.374306 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:34.374270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:34.374483 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.374465    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:34.374567 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.374487    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:34.374567 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.374501    2562 projected.go:196] Error preparing data for projected volume kube-api-access-x94c8 for pod openshift-network-diagnostics/network-check-target-27vsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:34.374567 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.374558    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8 podName:333a5c94-32b1-4b76-9e90-39206904ddce nodeName:}" failed. No retries permitted until 2026-09-10 23:36:38.374540518 +0000 UTC m=+10.205568055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x94c8" (UniqueName: "kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8") pod "network-check-target-27vsj" (UID: "333a5c94-32b1-4b76-9e90-39206904ddce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:34.651293 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:34.651172    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:34.651459 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.651317    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:34.651748 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:34.651721    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:34.651922 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.651831    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:34.652025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:34.651962    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:34.652086 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:34.652061    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:36.650596 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:36.650513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:36.650596 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:36.650561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:36.651121 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:36.650524    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:36.651121 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:36.650655    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:36.651121 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:36.650765    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:36.651121 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:36.650818    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:38.311263 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:38.310489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:38.311263 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:38.310553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:38.311263 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.310676    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:38.311263 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.310716    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:38.311263 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.310734    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret podName:3c15c11b-b1b2-4c3b-a4c8-edc70777af3d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:46.310715053 +0000 UTC m=+18.141742572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret") pod "global-pull-secret-syncer-qmbgq" (UID: "3c15c11b-b1b2-4c3b-a4c8-edc70777af3d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:38.311263 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.310781    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs podName:745744e6-2b42-4e82-bfba-6c2f79f57633 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:46.310761633 +0000 UTC m=+18.141789168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs") pod "network-metrics-daemon-m8gj9" (UID: "745744e6-2b42-4e82-bfba-6c2f79f57633") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:38.411378 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:38.411309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:38.411543 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.411473    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:38.411543 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.411495    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:38.411543 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.411508    2562 projected.go:196] Error preparing data for projected volume kube-api-access-x94c8 for pod openshift-network-diagnostics/network-check-target-27vsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:38.411731 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.411557    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8 podName:333a5c94-32b1-4b76-9e90-39206904ddce nodeName:}" failed. No retries permitted until 2026-09-10 23:36:46.411540333 +0000 UTC m=+18.242567849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x94c8" (UniqueName: "kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8") pod "network-check-target-27vsj" (UID: "333a5c94-32b1-4b76-9e90-39206904ddce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:38.651176 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:38.651085    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:38.651176 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:38.651107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:38.651374 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.651225    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:38.651374 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.651283    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:38.651374 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:38.651319    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:38.651508 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:38.651409    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:40.649889 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:40.649854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:40.650417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:40.649867    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:40.650417 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:40.649998    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:40.650417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:40.649868    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:40.650417 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:40.650071    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:40.650417 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:40.650133    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:42.649471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:42.649435    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:42.649471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:42.649465    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:42.649971 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:42.649435    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:42.649971 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:42.649563    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:42.649971 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:42.649648    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:42.649971 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:42.649745    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:44.649746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:44.649708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:44.649746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:44.649734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:44.650358 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:44.649838    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:44.650358 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:44.649884    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:44.650358 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:44.649999    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:44.650358 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:44.650067    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:46.374005 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:46.373968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:46.374627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:46.374023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:46.374627 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.374106    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:46.374627 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.374156    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:46.374627 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.374180    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs podName:745744e6-2b42-4e82-bfba-6c2f79f57633 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:02.374162985 +0000 UTC m=+34.205190496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs") pod "network-metrics-daemon-m8gj9" (UID: "745744e6-2b42-4e82-bfba-6c2f79f57633") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:46.374627 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.374207    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret podName:3c15c11b-b1b2-4c3b-a4c8-edc70777af3d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:02.374192327 +0000 UTC m=+34.205219933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret") pod "global-pull-secret-syncer-qmbgq" (UID: "3c15c11b-b1b2-4c3b-a4c8-edc70777af3d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:46.475142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:46.475109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:46.475315 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.475227    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:46.475315 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.475243    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:46.475315 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.475251    2562 projected.go:196] Error preparing data for projected volume kube-api-access-x94c8 for pod openshift-network-diagnostics/network-check-target-27vsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:46.475315 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.475299    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8 podName:333a5c94-32b1-4b76-9e90-39206904ddce nodeName:}" failed. No retries permitted until 2026-09-10 23:37:02.475286377 +0000 UTC m=+34.306313888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x94c8" (UniqueName: "kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8") pod "network-check-target-27vsj" (UID: "333a5c94-32b1-4b76-9e90-39206904ddce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:46.649526 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:46.649456    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:46.649526 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:46.649472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:46.649680 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:46.649464    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:46.649680 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.649566    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:46.649680 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.649629    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:46.649785 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:46.649719    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:48.650255 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:48.650217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:48.650712 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:48.650312    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:48.650712 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:48.650316    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:48.650712 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:48.650416    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:48.650712 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:48.650444    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:48.650712 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:48.650550    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:50.651583 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.649840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:50.651583 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:50.650261    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:50.651583 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.651342    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:50.651583 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:50.651436    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:50.651583 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.651462    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:50.651583 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:50.651582    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:50.832637 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.832485    2562 generic.go:356] "Generic (PLEG): container finished" podID="e48f204d-76c2-43fa-8100-486c6a3e31a3" containerID="b2bca6823b220a523c0315e724d4032d115473417ddfad2c480dd0e860e9e55d" exitCode=0
Sep 10 23:36:50.832792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.832565    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll288" event={"ID":"e48f204d-76c2-43fa-8100-486c6a3e31a3","Type":"ContainerDied","Data":"b2bca6823b220a523c0315e724d4032d115473417ddfad2c480dd0e860e9e55d"}
Sep 10 23:36:50.834384 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.834345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2lmmn" event={"ID":"2cf86d6f-6f67-40a4-98bf-f97c338816b3","Type":"ContainerStarted","Data":"7736b9487c8ec67525a9d510714654ff58c24ac769e6a761a3d1def2aed75f0f"}
Sep 10 23:36:50.835996 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.835964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gvg49" event={"ID":"1e324660-2423-4acb-becf-9a78be67198a","Type":"ContainerStarted","Data":"6fe84d8abfab0e42ae6897223623b0da96f53a9d72695838ae5607132858afde"}
Sep 10 23:36:50.837730 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.837706    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5" event={"ID":"2f1e94ed-d8d5-47da-8a95-072c5848b218","Type":"ContainerStarted","Data":"8b69ab372c1f66701f997f8b5961e190dbd7c327c36f539ca741490c5102fceb"}
Sep 10 23:36:50.839412 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.839387    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2q2tj" event={"ID":"6f736cd6-ad74-4550-a2a8-82f59ee9d98f","Type":"ContainerStarted","Data":"acfaafc766e51e2548ddcc72cfefc8a5e751762ad3a6f1d4b32e1aad6b62be4f"}
Sep 10 23:36:50.841059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.841034    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsf6w" event={"ID":"ad3a3d17-1e95-41bb-89a3-4f74e903e750","Type":"ContainerStarted","Data":"f55084f219df440a4826d22d25b4c2400a2075064dff6760314ff186b9210f66"}
Sep 10 23:36:50.842891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.842867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"50dc13baecc9f10bd590f1d4b64b29f74df1a3eb016a291a7954de38c16f5214"}
Sep 10 23:36:50.842989 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.842960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"3f76c11366d6804d48675c3fcad8043feb79d085299cc4db5b6276ada4cf18d3"}
Sep 10 23:36:50.844476 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.844458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7br8t" event={"ID":"d4962449-bcd3-4306-b62a-0de981ea1bdc","Type":"ContainerStarted","Data":"2e0b694a0c987f5d87e2963265e62919bf81fbe3ade75f7fedc1b578099c4bc2"}
Sep 10 23:36:50.856068 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.856031    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-218.ec2.internal" podStartSLOduration=20.856020533 podStartE2EDuration="20.856020533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:33.815822916 +0000 UTC m=+5.646850454" watchObservedRunningTime="2026-09-10 23:36:50.856020533 +0000 UTC m=+22.687048066"
Sep 10 23:36:50.871461 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.871423    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gvg49" podStartSLOduration=3.231632778 podStartE2EDuration="21.871410122s" totalImagesPullingTime="18.639777344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:50.871006286 +0000 UTC m=+22.702033845" watchObservedRunningTime="2026-09-10 23:36:50.871410122 +0000 UTC m=+22.702437652"
Sep 10 23:36:50.892349 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.892304    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2lmmn" podStartSLOduration=3.20822136 podStartE2EDuration="21.892291185s" totalImagesPullingTime="18.684069825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:50.891763116 +0000 UTC m=+22.722790650" watchObservedRunningTime="2026-09-10 23:36:50.892291185 +0000 UTC m=+22.723318718"
Sep 10 23:36:50.912535 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.912476    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zsf6w" podStartSLOduration=3.279477729 podStartE2EDuration="21.912460451s" totalImagesPullingTime="18.632982722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:50.9122695 +0000 UTC m=+22.743297035" watchObservedRunningTime="2026-09-10 23:36:50.912460451 +0000 UTC m=+22.743488064"
Sep 10 23:36:50.926539 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.926505    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7br8t" podStartSLOduration=3.299328149 podStartE2EDuration="21.926493486s" totalImagesPullingTime="18.627165337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:50.926406814 +0000 UTC m=+22.757434344" watchObservedRunningTime="2026-09-10 23:36:50.926493486 +0000 UTC m=+22.757521019"
Sep 10 23:36:50.944529 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.944496    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2q2tj" podStartSLOduration=3.30976964 podStartE2EDuration="21.944485532s" totalImagesPullingTime="18.634715892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:50.944079013 +0000 UTC m=+22.775106581" watchObservedRunningTime="2026-09-10 23:36:50.944485532 +0000 UTC m=+22.775513066"
Sep 10 23:36:50.965367 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.965345    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:50.965805 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:50.965790    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:51.849962 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:51.849859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"8392c2b757c5682d8519ee0a75cd3b3aee6e9fc968156bfe1a494f734a318186"}
Sep 10 23:36:51.850580 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:51.849964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"563e924dea2df0e0259d79252bb4093fb3a7b25f51861b091312a2739c83628a"}
Sep 10 23:36:51.850580 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:51.850019    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"b219d99f65de02d998f7fe7357648e0b393e17b535cd993b51cee2bf1470421d"}
Sep 10 23:36:51.850580 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:51.850066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"4c1ed8632aa6aecd975f749a44057e39cc79252c3e213b0fc50b8a67e790fbbd"}
Sep 10 23:36:52.025183 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.025154    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 23:36:52.606436 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.606321    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:36:52.025178063Z","UUID":"63e32326-0512-41e9-9851-c6a1e7a5ec78","Handler":null,"Name":"","Endpoint":""}
Sep 10 23:36:52.608025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.607991    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 23:36:52.608025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.608023    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 23:36:52.653440 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.653415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:52.653550 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.653421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:52.653550 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:52.653512    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:52.653641 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.653416    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:52.653696 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:52.653587    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:52.653696 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:52.653677    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:52.854267 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.854227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5" event={"ID":"2f1e94ed-d8d5-47da-8a95-072c5848b218","Type":"ContainerStarted","Data":"045eb4fd688655b5ec5da26e2151f36e2c91106995714a7d16624ec90c71a66f"}
Sep 10 23:36:52.856955 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.856885    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 23:36:52.856955 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.856885    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zj52b" event={"ID":"5224f0ca-4aee-4ea1-9c5e-6c8ae526192b","Type":"ContainerStarted","Data":"f16a44e77568518e192b1a2b5320113fc36288d69aa108a8aba169ef94bba148"}
Sep 10 23:36:52.872555 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:52.872466    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zj52b" podStartSLOduration=5.219423078 podStartE2EDuration="23.872449425s" totalImagesPullingTime="18.653026347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:52.871971568 +0000 UTC m=+24.702999117" watchObservedRunningTime="2026-09-10 23:36:52.872449425 +0000 UTC m=+24.703476959"
Sep 10 23:36:53.861170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:53.861138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5" event={"ID":"2f1e94ed-d8d5-47da-8a95-072c5848b218","Type":"ContainerStarted","Data":"c72485f33ae2ddb90ec65d9ceddf8b89783105c1975b14fb00d22c903d5c6b8a"}
Sep 10 23:36:54.650138 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:54.650102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:54.650311 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:54.650103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:54.650311 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:54.650225    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:54.650311 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:54.650257    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:54.650459 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:54.650380    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:54.650513 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:54.650461    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:55.631943 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:55.631746    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:55.632355 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:55.632044    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 23:36:55.632474 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:55.632453    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7br8t"
Sep 10 23:36:55.646624 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:55.646591    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j92r5" podStartSLOduration=5.838450553 podStartE2EDuration="26.64657733s" totalImagesPullingTime="20.808126777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:53.89029157 +0000 UTC m=+25.721319124" watchObservedRunningTime="2026-09-10 23:36:55.64657733 +0000 UTC m=+27.477604923"
Sep 10 23:36:55.868097 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:55.868073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"2df6601a0fee11d027464b358b1789cbfd4f6a342007112b8805452c1f03ad95"}
Sep 10 23:36:55.869624 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:55.869606    2562 generic.go:356] "Generic (PLEG): container finished" podID="e48f204d-76c2-43fa-8100-486c6a3e31a3" containerID="d6eb1cc62a82d334b4b7deb970aa00b78bbefa281fa45c07033ac8cf8d0e5235" exitCode=0
Sep 10 23:36:55.869709 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:55.869673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll288" event={"ID":"e48f204d-76c2-43fa-8100-486c6a3e31a3","Type":"ContainerDied","Data":"d6eb1cc62a82d334b4b7deb970aa00b78bbefa281fa45c07033ac8cf8d0e5235"}
Sep 10 23:36:56.649357 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:56.649326    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:56.649357 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:56.649345    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:56.649693 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:56.649326    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:56.649693 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:56.649442    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:56.649693 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:56.649529    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:56.649693 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:56.649639    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:56.873073 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:56.873050    2562 generic.go:356] "Generic (PLEG): container finished" podID="e48f204d-76c2-43fa-8100-486c6a3e31a3" containerID="30bc09904b0d0c8290be6106d878345ecc53e0ccedaf36972c93a5aba0d7858f" exitCode=0
Sep 10 23:36:56.873180 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:56.873101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll288" event={"ID":"e48f204d-76c2-43fa-8100-486c6a3e31a3","Type":"ContainerDied","Data":"30bc09904b0d0c8290be6106d878345ecc53e0ccedaf36972c93a5aba0d7858f"}
Sep 10 23:36:57.878447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.878187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" event={"ID":"fc61cf8b-3a02-49fd-b1f5-92ddd99729ac","Type":"ContainerStarted","Data":"cd9f347046bd6c8f178b6cc0ec9e2de684325062b3e36c4d8c60305a1f291fcd"}
Sep 10 23:36:57.878857 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.878472    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:57.878857 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.878531    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:57.878857 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.878579    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:57.880991 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.880965    2562 generic.go:356] "Generic (PLEG): container finished" podID="e48f204d-76c2-43fa-8100-486c6a3e31a3" containerID="65d4d3b331e0212a6d8c23057f95ac488568dcdb7c4175be792ec04dfd268dbf" exitCode=0
Sep 10 23:36:57.881102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.881009    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll288" event={"ID":"e48f204d-76c2-43fa-8100-486c6a3e31a3","Type":"ContainerDied","Data":"65d4d3b331e0212a6d8c23057f95ac488568dcdb7c4175be792ec04dfd268dbf"}
Sep 10 23:36:57.893713 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.893695    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:57.893787 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.893782    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:36:57.908470 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:57.908433    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7" podStartSLOduration=10.211335335 podStartE2EDuration="28.908422066s" totalImagesPullingTime="18.697086731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:57.907880846 +0000 UTC m=+29.738908405" watchObservedRunningTime="2026-09-10 23:36:57.908422066 +0000 UTC m=+29.739449599"
Sep 10 23:36:58.650737 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.650663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:58.650876 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.650756    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:58.650876 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.650801    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:58.650876 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:58.650820    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:58.651057 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:58.650888    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:36:58.651057 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:58.650991    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:58.858100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.858071    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-27vsj"]
Sep 10 23:36:58.862287 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.860796    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qmbgq"]
Sep 10 23:36:58.875188 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.875164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m8gj9"]
Sep 10 23:36:58.883339 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.883318    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:36:58.884056 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.883336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:36:58.884056 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:36:58.883319    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:36:58.884056 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:58.883422    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:36:58.884056 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:58.883893    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:36:58.884056 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:36:58.884003    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:37:00.649784 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:00.649752    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:37:00.649784 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:00.649772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:37:00.650460 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:00.649867    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:37:00.650460 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:00.649988    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:37:00.650460 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:00.650023    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:37:00.650460 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:00.650125    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:37:02.397005 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:02.396802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:37:02.397305 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:02.397033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:37:02.397305 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.396912    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:37:02.397305 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.397128    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs podName:745744e6-2b42-4e82-bfba-6c2f79f57633 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:34.397111968 +0000 UTC m=+66.228139479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs") pod "network-metrics-daemon-m8gj9" (UID: "745744e6-2b42-4e82-bfba-6c2f79f57633") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:37:02.397305 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.397130    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:37:02.397305 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.397167    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret podName:3c15c11b-b1b2-4c3b-a4c8-edc70777af3d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:34.39715671 +0000 UTC m=+66.228184221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret") pod "global-pull-secret-syncer-qmbgq" (UID: "3c15c11b-b1b2-4c3b-a4c8-edc70777af3d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:37:02.498188 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:02.498122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:37:02.498301 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.498249    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:37:02.498301 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.498264    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:37:02.498301 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.498272    2562 projected.go:196] Error preparing data for projected volume kube-api-access-x94c8 for pod openshift-network-diagnostics/network-check-target-27vsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:37:02.498399 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.498315    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8 podName:333a5c94-32b1-4b76-9e90-39206904ddce nodeName:}" failed. No retries permitted until 2026-09-10 23:37:34.498302519 +0000 UTC m=+66.329330030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x94c8" (UniqueName: "kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8") pod "network-check-target-27vsj" (UID: "333a5c94-32b1-4b76-9e90-39206904ddce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:37:02.649580 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:02.649545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:37:02.649580 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:02.649574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:37:02.649763 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:02.649545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:37:02.649763 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.649649    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-27vsj" podUID="333a5c94-32b1-4b76-9e90-39206904ddce"
Sep 10 23:37:02.649763 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.649728    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8gj9" podUID="745744e6-2b42-4e82-bfba-6c2f79f57633"
Sep 10 23:37:02.649871 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:02.649806    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbgq" podUID="3c15c11b-b1b2-4c3b-a4c8-edc70777af3d"
Sep 10 23:37:02.894807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:02.894732    2562 generic.go:356] "Generic (PLEG): container finished" podID="e48f204d-76c2-43fa-8100-486c6a3e31a3" containerID="8c3010d2d4c2e7f1165d4b1d1d2a6834ea973ba2d5f7d4341537851275f793d1" exitCode=0
Sep 10 23:37:02.894807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:02.894794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll288" event={"ID":"e48f204d-76c2-43fa-8100-486c6a3e31a3","Type":"ContainerDied","Data":"8c3010d2d4c2e7f1165d4b1d1d2a6834ea973ba2d5f7d4341537851275f793d1"}
Sep 10 23:37:03.484674 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.484650    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-218.ec2.internal" event="NodeReady"
Sep 10 23:37:03.485121 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.484743    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 23:37:03.522050 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.522025    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"]
Sep 10 23:37:03.545179 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.545157    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"]
Sep 10 23:37:03.545303 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.545188    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"]
Sep 10 23:37:03.545382 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.545300    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.550834 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.550817    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 23:37:03.551219 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.551198    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 23:37:03.551720 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.551706    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wsf9n\""
Sep 10 23:37:03.556667 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.556648    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 23:37:03.557450 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.557436    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 23:37:03.569130 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.569109    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gw9pf"]
Sep 10 23:37:03.569256 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.569235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.572828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.572707    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 23:37:03.572828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.572720    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fdx7s\""
Sep 10 23:37:03.573027 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.573012    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.573093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.573066    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 23:37:03.573152 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.573101    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.593232 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.593190    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr"]
Sep 10 23:37:03.593360 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.593340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.597169 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.596558    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 23:37:03.597169 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.596618    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-v7m82\""
Sep 10 23:37:03.597169 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.596906    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 23:37:03.597169 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.596939    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.600565 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.600469    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.604705 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.604683    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 23:37:03.609211 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.609189    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-wdgcw"]
Sep 10 23:37:03.609324 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.609309    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr"
Sep 10 23:37:03.611937 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.611907    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5829z\""
Sep 10 23:37:03.612013 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.611962    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.612013 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.611997    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.631434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.631416    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"]
Sep 10 23:37:03.631548 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.631533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.636320 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.636303    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 23:37:03.636416 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.636371    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.636675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.636658    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nl66z\""
Sep 10 23:37:03.636675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.636671    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 23:37:03.636778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.636658    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.643065 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.643041    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 23:37:03.651412 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.651396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"]
Sep 10 23:37:03.651488 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.651420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-wdgcw"]
Sep 10 23:37:03.651488 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.651433    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gw9pf"]
Sep 10 23:37:03.651488 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.651445    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr"]
Sep 10 23:37:03.651488 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.651468    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"]
Sep 10 23:37:03.651606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.651519    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.651606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.651564    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2jz8n"]
Sep 10 23:37:03.655769 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.655749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 23:37:03.655880 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.655779    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 23:37:03.656255 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.656237    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.656340 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.656238    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8n4tg\""
Sep 10 23:37:03.656340 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.656274    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.665179 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.665164    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"]
Sep 10 23:37:03.665270 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.665257    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.668045 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.668031    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9sdk\""
Sep 10 23:37:03.668115 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.668028    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 23:37:03.669069 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.669051    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 23:37:03.687190 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.687170    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"]
Sep 10 23:37:03.687299 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.687285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:03.689950 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.689918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 23:37:03.690028 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.689954    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.690448 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.690431    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7gx8t\""
Sep 10 23:37:03.690530 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.690516    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.705482 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/576b260f-d773-4bd1-b98b-fdc0d99626d5-ca-trust-extracted\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.705563 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-certificates\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.705563 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43537109-e795-4d12-9269-b5578783dc5e-serving-cert\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.705563 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2ts\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-kube-api-access-qh2ts\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.705563 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43537109-e795-4d12-9269-b5578783dc5e-trusted-ca\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.705732 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-installation-pull-secrets\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.705732 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7hc\" (UniqueName: \"kubernetes.io/projected/96674fff-8032-41fb-b75e-ca10f7d60860-kube-api-access-vv7hc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.705732 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv2t2\" (UniqueName: \"kubernetes.io/projected/a93fdd16-d399-4023-810e-d611f8f7aee0-kube-api-access-lv2t2\") pod \"volume-data-source-validator-86695b7cc7-r9bdr\" (UID: \"a93fdd16-d399-4023-810e-d611f8f7aee0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr"
Sep 10 23:37:03.705732 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-image-registry-private-configuration\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.705732 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96674fff-8032-41fb-b75e-ca10f7d60860-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.705986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96674fff-8032-41fb-b75e-ca10f7d60860-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.705986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-trusted-ca\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.705986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-bound-sa-token\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.705986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.705986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705888    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43537109-e795-4d12-9269-b5578783dc5e-config\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.705986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.705911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqktc\" (UniqueName: \"kubernetes.io/projected/43537109-e795-4d12-9269-b5578783dc5e-kube-api-access-nqktc\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.711231 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.711214    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c5d97497d-trwfb"]
Sep 10 23:37:03.711331 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.711315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.714105 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.714084    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bddzj\""
Sep 10 23:37:03.714195 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.714137    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.714195 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.714145    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.714732 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.714714    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 23:37:03.714963 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.714948    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 23:37:03.735286 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.735242    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jvt2x"]
Sep 10 23:37:03.735373 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.735344    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:03.738551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.738532    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 23:37:03.738645 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.738536    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 23:37:03.738645 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.738563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 23:37:03.738645 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.738584    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.738645 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.738638    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 23:37:03.738893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.738780    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4hzqj\""
Sep 10 23:37:03.738893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.738850    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.759650 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.759631    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5"]
Sep 10 23:37:03.759735 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.759721    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jvt2x"
Sep 10 23:37:03.762179 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.762163    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.762314 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.762285    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qfssj\""
Sep 10 23:37:03.762381 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.762285    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 23:37:03.762432 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.762407    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.769921 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.769883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:03.786401 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.786386    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2jz8n"]
Sep 10 23:37:03.786501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.786415    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5"]
Sep 10 23:37:03.786501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.786429    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"]
Sep 10 23:37:03.786501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.786445    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"]
Sep 10 23:37:03.786501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.786456    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c5d97497d-trwfb"]
Sep 10 23:37:03.786501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.786469    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jvt2x"]
Sep 10 23:37:03.786501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.786497    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5"
Sep 10 23:37:03.789948 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.789900    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:37:03.789948 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.789906    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xbzts\""
Sep 10 23:37:03.790090 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.789973    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:37:03.806503 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/576b260f-d773-4bd1-b98b-fdc0d99626d5-ca-trust-extracted\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.806571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-certificates\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.806571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/356212ea-1e0e-452a-8e4c-a270b6c46ff7-serving-cert\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.806571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43537109-e795-4d12-9269-b5578783dc5e-trusted-ca\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.806571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806561    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c84d6c-7d15-4b25-be27-78c331ceb2ae-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.806761 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806610    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.806761 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-installation-pull-secrets\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.806761 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb44j\" (UniqueName: \"kubernetes.io/projected/915d900b-a1a6-428b-ac3a-0641215f0233-kube-api-access-nb44j\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.806761 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65f4n\" (UniqueName: \"kubernetes.io/projected/356212ea-1e0e-452a-8e4c-a270b6c46ff7-kube-api-access-65f4n\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.806761 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv2t2\" (UniqueName: \"kubernetes.io/projected/a93fdd16-d399-4023-810e-d611f8f7aee0-kube-api-access-lv2t2\") pod \"volume-data-source-validator-86695b7cc7-r9bdr\" (UID: \"a93fdd16-d399-4023-810e-d611f8f7aee0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr"
Sep 10 23:37:03.806761 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56q4b\" (UniqueName: \"kubernetes.io/projected/b472852f-5f8d-4d93-9d0d-45fb4a24f564-kube-api-access-56q4b\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c84d6c-7d15-4b25-be27-78c331ceb2ae-tmp\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96674fff-8032-41fb-b75e-ca10f7d60860-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806861    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-trusted-ca\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-bound-sa-token\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/576b260f-d773-4bd1-b98b-fdc0d99626d5-ca-trust-extracted\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43537109-e795-4d12-9269-b5578783dc5e-config\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.806992    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17c84d6c-7d15-4b25-be27-78c331ceb2ae-snapshots\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks522\" (UniqueName: \"kubernetes.io/projected/17c84d6c-7d15-4b25-be27-78c331ceb2ae-kube-api-access-ks522\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.807070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/356212ea-1e0e-452a-8e4c-a270b6c46ff7-config\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c84d6c-7d15-4b25-be27-78c331ceb2ae-service-ca-bundle\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/915d900b-a1a6-428b-ac3a-0641215f0233-config-volume\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915d900b-a1a6-428b-ac3a-0641215f0233-tmp-dir\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807125    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-certificates\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2ts\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-kube-api-access-qh2ts\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz7bp\" (UniqueName: \"kubernetes.io/projected/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-kube-api-access-bz7bp\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17c84d6c-7d15-4b25-be27-78c331ceb2ae-serving-cert\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7hc\" (UniqueName: \"kubernetes.io/projected/96674fff-8032-41fb-b75e-ca10f7d60860-kube-api-access-vv7hc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-image-registry-private-configuration\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96674fff-8032-41fb-b75e-ca10f7d60860-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43537109-e795-4d12-9269-b5578783dc5e-serving-cert\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.807510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqktc\" (UniqueName: \"kubernetes.io/projected/43537109-e795-4d12-9269-b5578783dc5e-kube-api-access-nqktc\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.808242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:03.808242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96674fff-8032-41fb-b75e-ca10f7d60860-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.808242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807552    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43537109-e795-4d12-9269-b5578783dc5e-config\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.808242 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.807714    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:03.808242 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.807729    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6c8fc5d7-jkp94: secret "image-registry-tls" not found
Sep 10 23:37:03.808242 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.807783    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls podName:576b260f-d773-4bd1-b98b-fdc0d99626d5 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.307766389 +0000 UTC m=+36.138793903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls") pod "image-registry-5c6c8fc5d7-jkp94" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5") : secret "image-registry-tls" not found
Sep 10 23:37:03.808242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.807965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-trusted-ca\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.808242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.808183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43537109-e795-4d12-9269-b5578783dc5e-trusted-ca\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.810826 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.810802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96674fff-8032-41fb-b75e-ca10f7d60860-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.810956 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.810918    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43537109-e795-4d12-9269-b5578783dc5e-serving-cert\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.811074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.811029    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-installation-pull-secrets\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.811074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.811040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-image-registry-private-configuration\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.818733 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.818705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2ts\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-kube-api-access-qh2ts\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.818812 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.818751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv2t2\" (UniqueName: \"kubernetes.io/projected/a93fdd16-d399-4023-810e-d611f8f7aee0-kube-api-access-lv2t2\") pod \"volume-data-source-validator-86695b7cc7-r9bdr\" (UID: \"a93fdd16-d399-4023-810e-d611f8f7aee0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr"
Sep 10 23:37:03.819163 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.819139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7hc\" (UniqueName: \"kubernetes.io/projected/96674fff-8032-41fb-b75e-ca10f7d60860-kube-api-access-vv7hc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69fsw\" (UID: \"96674fff-8032-41fb-b75e-ca10f7d60860\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.819316 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.819299    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-bound-sa-token\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:03.819736 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.819717    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqktc\" (UniqueName: \"kubernetes.io/projected/43537109-e795-4d12-9269-b5578783dc5e-kube-api-access-nqktc\") pod \"console-operator-696884fb9c-gw9pf\" (UID: \"43537109-e795-4d12-9269-b5578783dc5e\") " pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.878837 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.878818    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"
Sep 10 23:37:03.897104 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.897084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:03.900367 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.900346    2562 generic.go:356] "Generic (PLEG): container finished" podID="e48f204d-76c2-43fa-8100-486c6a3e31a3" containerID="7dadec941be4be183e3f30224b14907c370f7215b009eac51c043b686fa62e27" exitCode=0
Sep 10 23:37:03.900448 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.900404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll288" event={"ID":"e48f204d-76c2-43fa-8100-486c6a3e31a3","Type":"ContainerDied","Data":"7dadec941be4be183e3f30224b14907c370f7215b009eac51c043b686fa62e27"}
Sep 10 23:37:03.907501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.907482    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:03.908660 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17c84d6c-7d15-4b25-be27-78c331ceb2ae-snapshots\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.908758 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks522\" (UniqueName: \"kubernetes.io/projected/17c84d6c-7d15-4b25-be27-78c331ceb2ae-kube-api-access-ks522\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.908758 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/356212ea-1e0e-452a-8e4c-a270b6c46ff7-config\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.908758 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c84d6c-7d15-4b25-be27-78c331ceb2ae-service-ca-bundle\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.908758 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/915d900b-a1a6-428b-ac3a-0641215f0233-config-volume\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.908972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908779    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915d900b-a1a6-428b-ac3a-0641215f0233-tmp-dir\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.908972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.908972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh5wg\" (UniqueName: \"kubernetes.io/projected/156442c4-f848-4a58-899f-26ea730df356-kube-api-access-mh5wg\") pod \"ingress-canary-jvt2x\" (UID: \"156442c4-f848-4a58-899f-26ea730df356\") " pod="openshift-ingress-canary/ingress-canary-jvt2x"
Sep 10 23:37:03.908972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz7bp\" (UniqueName: \"kubernetes.io/projected/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-kube-api-access-bz7bp\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.908972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:03.908972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.908986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17c84d6c-7d15-4b25-be27-78c331ceb2ae-serving-cert\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlx27\" (UniqueName: \"kubernetes.io/projected/12bf9c01-3b14-47e8-97a1-cd867914835f-kube-api-access-hlx27\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4fj\" (UniqueName: \"kubernetes.io/projected/4ab50fa8-0838-473c-85e9-11dfd2ebdc35-kube-api-access-fx4fj\") pod \"network-check-source-6f8b768455-mdqv5\" (UID: \"4ab50fa8-0838-473c-85e9-11dfd2ebdc35\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915d900b-a1a6-428b-ac3a-0641215f0233-tmp-dir\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-default-certificate\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909199    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-stats-auth\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:03.909247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/356212ea-1e0e-452a-8e4c-a270b6c46ff7-serving-cert\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c84d6c-7d15-4b25-be27-78c331ceb2ae-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/356212ea-1e0e-452a-8e4c-a270b6c46ff7-config\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17c84d6c-7d15-4b25-be27-78c331ceb2ae-snapshots\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb44j\" (UniqueName: \"kubernetes.io/projected/915d900b-a1a6-428b-ac3a-0641215f0233-kube-api-access-nb44j\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65f4n\" (UniqueName: \"kubernetes.io/projected/356212ea-1e0e-452a-8e4c-a270b6c46ff7-kube-api-access-65f4n\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c84d6c-7d15-4b25-be27-78c331ceb2ae-service-ca-bundle\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56q4b\" (UniqueName: \"kubernetes.io/projected/b472852f-5f8d-4d93-9d0d-45fb4a24f564-kube-api-access-56q4b\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/156442c4-f848-4a58-899f-26ea730df356-cert\") pod \"ingress-canary-jvt2x\" (UID: \"156442c4-f848-4a58-899f-26ea730df356\") " pod="openshift-ingress-canary/ingress-canary-jvt2x"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c84d6c-7d15-4b25-be27-78c331ceb2ae-tmp\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/915d900b-a1a6-428b-ac3a-0641215f0233-config-volume\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.909719 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.909711    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:03.910294 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.909769    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls podName:661b8c3f-b66f-4989-ae4d-2ceb5e3b831f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.409752687 +0000 UTC m=+36.240780227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n86rk" (UID: "661b8c3f-b66f-4989-ae4d-2ceb5e3b831f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:03.910294 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.909785    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:03.910294 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c84d6c-7d15-4b25-be27-78c331ceb2ae-tmp\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.910294 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.909820    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls podName:915d900b-a1a6-428b-ac3a-0641215f0233 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.409808812 +0000 UTC m=+36.240836325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls") pod "dns-default-2jz8n" (UID: "915d900b-a1a6-428b-ac3a-0641215f0233") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:03.910294 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.909841    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:03.910294 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:03.909875    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls podName:b472852f-5f8d-4d93-9d0d-45fb4a24f564 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.409862908 +0000 UTC m=+36.240890450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9b7h5" (UID: "b472852f-5f8d-4d93-9d0d-45fb4a24f564") : secret "samples-operator-tls" not found
Sep 10 23:37:03.910294 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.909965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.910795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.910760    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c84d6c-7d15-4b25-be27-78c331ceb2ae-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.912349 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.912320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/356212ea-1e0e-452a-8e4c-a270b6c46ff7-serving-cert\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.912427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.912371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17c84d6c-7d15-4b25-be27-78c331ceb2ae-serving-cert\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.916821 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.916807    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr"
Sep 10 23:37:03.918963 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.918945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb44j\" (UniqueName: \"kubernetes.io/projected/915d900b-a1a6-428b-ac3a-0641215f0233-kube-api-access-nb44j\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:03.919871 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.919846    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks522\" (UniqueName: \"kubernetes.io/projected/17c84d6c-7d15-4b25-be27-78c331ceb2ae-kube-api-access-ks522\") pod \"insights-operator-5844b856db-wdgcw\" (UID: \"17c84d6c-7d15-4b25-be27-78c331ceb2ae\") " pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:03.921377 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.921339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz7bp\" (UniqueName: \"kubernetes.io/projected/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-kube-api-access-bz7bp\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:03.926003 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.925972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65f4n\" (UniqueName: \"kubernetes.io/projected/356212ea-1e0e-452a-8e4c-a270b6c46ff7-kube-api-access-65f4n\") pod \"service-ca-operator-cf7c776b7-hgt5s\" (UID: \"356212ea-1e0e-452a-8e4c-a270b6c46ff7\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:03.926683 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.926659    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56q4b\" (UniqueName: \"kubernetes.io/projected/b472852f-5f8d-4d93-9d0d-45fb4a24f564-kube-api-access-56q4b\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:03.938884 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:03.938855    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-wdgcw"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.010090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-default-certificate\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.010325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-stats-auth\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.010393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/156442c4-f848-4a58-899f-26ea730df356-cert\") pod \"ingress-canary-jvt2x\" (UID: \"156442c4-f848-4a58-899f-26ea730df356\") " pod="openshift-ingress-canary/ingress-canary-jvt2x"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.010501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5wg\" (UniqueName: \"kubernetes.io/projected/156442c4-f848-4a58-899f-26ea730df356-kube-api-access-mh5wg\") pod \"ingress-canary-jvt2x\" (UID: \"156442c4-f848-4a58-899f-26ea730df356\") " pod="openshift-ingress-canary/ingress-canary-jvt2x"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.010524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.010543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.010579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlx27\" (UniqueName: \"kubernetes.io/projected/12bf9c01-3b14-47e8-97a1-cd867914835f-kube-api-access-hlx27\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.010622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4fj\" (UniqueName: \"kubernetes.io/projected/4ab50fa8-0838-473c-85e9-11dfd2ebdc35-kube-api-access-fx4fj\") pod \"network-check-source-6f8b768455-mdqv5\" (UID: \"4ab50fa8-0838-473c-85e9-11dfd2ebdc35\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5"
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.012106    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.012180    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.512160191 +0000 UTC m=+36.343187721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : secret "router-metrics-certs-default" not found
Sep 10 23:37:04.012746 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.012576    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:04.512558096 +0000 UTC m=+36.343585624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:04.017942 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.017906    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/156442c4-f848-4a58-899f-26ea730df356-cert\") pod \"ingress-canary-jvt2x\" (UID: \"156442c4-f848-4a58-899f-26ea730df356\") " pod="openshift-ingress-canary/ingress-canary-jvt2x"
Sep 10 23:37:04.018663 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.018383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"
Sep 10 23:37:04.018663 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.018614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-stats-auth\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.022696 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.019268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-default-certificate\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.022696 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.020551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5wg\" (UniqueName: \"kubernetes.io/projected/156442c4-f848-4a58-899f-26ea730df356-kube-api-access-mh5wg\") pod \"ingress-canary-jvt2x\" (UID: \"156442c4-f848-4a58-899f-26ea730df356\") " pod="openshift-ingress-canary/ingress-canary-jvt2x"
Sep 10 23:37:04.023344 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.023318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4fj\" (UniqueName: \"kubernetes.io/projected/4ab50fa8-0838-473c-85e9-11dfd2ebdc35-kube-api-access-fx4fj\") pod \"network-check-source-6f8b768455-mdqv5\" (UID: \"4ab50fa8-0838-473c-85e9-11dfd2ebdc35\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5"
Sep 10 23:37:04.024963 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.024770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlx27\" (UniqueName: \"kubernetes.io/projected/12bf9c01-3b14-47e8-97a1-cd867914835f-kube-api-access-hlx27\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.067505 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.067482    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jvt2x"
Sep 10 23:37:04.094361 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.094340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5"
Sep 10 23:37:04.218873 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.218849    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw"]
Sep 10 23:37:04.223992 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.223491    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-gw9pf"]
Sep 10 23:37:04.227176 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:04.227154    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43537109_e795_4d12_9269_b5578783dc5e.slice/crio-b32db31c8d6658033957fa4463b7f94beaeb46fde441c789f823f34726648328 WatchSource:0}: Error finding container b32db31c8d6658033957fa4463b7f94beaeb46fde441c789f823f34726648328: Status 404 returned error can't find the container with id b32db31c8d6658033957fa4463b7f94beaeb46fde441c789f823f34726648328
Sep 10 23:37:04.235480 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.235435    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr"]
Sep 10 23:37:04.237537 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:04.237518    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93fdd16_d399_4023_810e_d611f8f7aee0.slice/crio-b30b3724087d534925e7bf9351b685ca913aa16621c0f4de501d6375e43ab30d WatchSource:0}: Error finding container b30b3724087d534925e7bf9351b685ca913aa16621c0f4de501d6375e43ab30d: Status 404 returned error can't find the container with id b30b3724087d534925e7bf9351b685ca913aa16621c0f4de501d6375e43ab30d
Sep 10 23:37:04.265306 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:04.265287    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c84d6c_7d15_4b25_be27_78c331ceb2ae.slice/crio-4957a0fee9f621d8c8cf28a5c7280136f1b5a2359166b1e9004997c56c64492a WatchSource:0}: Error finding container 4957a0fee9f621d8c8cf28a5c7280136f1b5a2359166b1e9004997c56c64492a: Status 404 returned error can't find the container with id 4957a0fee9f621d8c8cf28a5c7280136f1b5a2359166b1e9004997c56c64492a
Sep 10 23:37:04.271309 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.271289    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-wdgcw"]
Sep 10 23:37:04.310665 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.310622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s"]
Sep 10 23:37:04.312530 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:04.312508    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356212ea_1e0e_452a_8e4c_a270b6c46ff7.slice/crio-6be8f38b02e449845b966964c883b877caf79f4fd1d3414b201f5bdfbd61932e WatchSource:0}: Error finding container 6be8f38b02e449845b966964c883b877caf79f4fd1d3414b201f5bdfbd61932e: Status 404 returned error can't find the container with id 6be8f38b02e449845b966964c883b877caf79f4fd1d3414b201f5bdfbd61932e
Sep 10 23:37:04.312736 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.312716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:04.312863 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.312851    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:04.312916 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.312864    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6c8fc5d7-jkp94: secret "image-registry-tls" not found
Sep 10 23:37:04.312916 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.312904    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls podName:576b260f-d773-4bd1-b98b-fdc0d99626d5 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:05.312891346 +0000 UTC m=+37.143918858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls") pod "image-registry-5c6c8fc5d7-jkp94" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5") : secret "image-registry-tls" not found
Sep 10 23:37:04.370218 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.370191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jvt2x"]
Sep 10 23:37:04.382742 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:04.382719    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156442c4_f848_4a58_899f_26ea730df356.slice/crio-d63ce05cce0fa5e929e5b5cfcdb6465efc53052916f2b3c24f6555bb37645280 WatchSource:0}: Error finding container d63ce05cce0fa5e929e5b5cfcdb6465efc53052916f2b3c24f6555bb37645280: Status 404 returned error can't find the container with id d63ce05cce0fa5e929e5b5cfcdb6465efc53052916f2b3c24f6555bb37645280
Sep 10 23:37:04.386746 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.386725    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5"]
Sep 10 23:37:04.389263 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:04.389241    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab50fa8_0838_473c_85e9_11dfd2ebdc35.slice/crio-5bdd7a1d364cc6267f05553a6cd675ed1fe6978a1d8f43f3e3bf2039b99c9bdb WatchSource:0}: Error finding container 5bdd7a1d364cc6267f05553a6cd675ed1fe6978a1d8f43f3e3bf2039b99c9bdb: Status 404 returned error can't find the container with id 5bdd7a1d364cc6267f05553a6cd675ed1fe6978a1d8f43f3e3bf2039b99c9bdb
Sep 10 23:37:04.413990 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.413971    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:04.414082 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.414010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:04.414082 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.414079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:04.414185 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.414105    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:04.414185 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.414156    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls podName:661b8c3f-b66f-4989-ae4d-2ceb5e3b831f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:05.414142291 +0000 UTC m=+37.245169831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n86rk" (UID: "661b8c3f-b66f-4989-ae4d-2ceb5e3b831f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:04.414185 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.414155    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:04.414185 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.414181    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls podName:915d900b-a1a6-428b-ac3a-0641215f0233 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:05.414174966 +0000 UTC m=+37.245202477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls") pod "dns-default-2jz8n" (UID: "915d900b-a1a6-428b-ac3a-0641215f0233") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:04.414344 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.414184    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:04.414344 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.414243    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls podName:b472852f-5f8d-4d93-9d0d-45fb4a24f564 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:05.414223575 +0000 UTC m=+37.245251101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9b7h5" (UID: "b472852f-5f8d-4d93-9d0d-45fb4a24f564") : secret "samples-operator-tls" not found
Sep 10 23:37:04.514794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.514743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.514794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.514772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:04.515331 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.514870    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:04.515331 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.514893    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:05.514875762 +0000 UTC m=+37.345903276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:04.515331 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:04.514975    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:05.51496724 +0000 UTC m=+37.345994750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : secret "router-metrics-certs-default" not found
Sep 10 23:37:04.649875 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.649849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:37:04.649875 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.649880    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:37:04.650085 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.649840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:37:04.652976 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.652775    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ppbqc\""
Sep 10 23:37:04.652976 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.652809    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n6qg4\""
Sep 10 23:37:04.652976 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.652839    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:37:04.652976 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.652783    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:37:04.907143 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.907008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s" event={"ID":"356212ea-1e0e-452a-8e4c-a270b6c46ff7","Type":"ContainerStarted","Data":"6be8f38b02e449845b966964c883b877caf79f4fd1d3414b201f5bdfbd61932e"}
Sep 10 23:37:04.909157 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.909099    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr" event={"ID":"a93fdd16-d399-4023-810e-d611f8f7aee0","Type":"ContainerStarted","Data":"b30b3724087d534925e7bf9351b685ca913aa16621c0f4de501d6375e43ab30d"}
Sep 10 23:37:04.915218 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.914283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll288" event={"ID":"e48f204d-76c2-43fa-8100-486c6a3e31a3","Type":"ContainerStarted","Data":"b8381b03d95c404dafa382711a9ec31559fcb3139c716d0e5830079ee6367deb"}
Sep 10 23:37:04.920111 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.918528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5" event={"ID":"4ab50fa8-0838-473c-85e9-11dfd2ebdc35","Type":"ContainerStarted","Data":"5bdd7a1d364cc6267f05553a6cd675ed1fe6978a1d8f43f3e3bf2039b99c9bdb"}
Sep 10 23:37:04.923979 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.923581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jvt2x" event={"ID":"156442c4-f848-4a58-899f-26ea730df356","Type":"ContainerStarted","Data":"d63ce05cce0fa5e929e5b5cfcdb6465efc53052916f2b3c24f6555bb37645280"}
Sep 10 23:37:04.927087 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.927016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw" event={"ID":"96674fff-8032-41fb-b75e-ca10f7d60860","Type":"ContainerStarted","Data":"5cc03f82a214aafee031cf97a1a4f39418b97ecf6ed36454dc27916e51c5f1a8"}
Sep 10 23:37:04.930020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.929957    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wdgcw" event={"ID":"17c84d6c-7d15-4b25-be27-78c331ceb2ae","Type":"ContainerStarted","Data":"4957a0fee9f621d8c8cf28a5c7280136f1b5a2359166b1e9004997c56c64492a"}
Sep 10 23:37:04.932209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.932153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf" event={"ID":"43537109-e795-4d12-9269-b5578783dc5e","Type":"ContainerStarted","Data":"b32db31c8d6658033957fa4463b7f94beaeb46fde441c789f823f34726648328"}
Sep 10 23:37:04.946581 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:04.945672    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ll288" podStartSLOduration=7.6569140959999995 podStartE2EDuration="35.945653767s" totalImagesPullingTime="28.288739671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:04.943983312 +0000 UTC m=+36.775010866" watchObservedRunningTime="2026-09-10 23:37:04.945653767 +0000 UTC m=+36.776681302"
Sep 10 23:37:05.078212 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:05.078116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:05.180611 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:05.180451    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:05.323481 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:05.323444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:05.323949 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.323724    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:05.323949 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.323748    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6c8fc5d7-jkp94: secret "image-registry-tls" not found
Sep 10 23:37:05.323949 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.323811    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls podName:576b260f-d773-4bd1-b98b-fdc0d99626d5 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.323792052 +0000 UTC m=+39.154819568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls") pod "image-registry-5c6c8fc5d7-jkp94" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5") : secret "image-registry-tls" not found
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:05.424165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:05.424252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:05.424302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.424480    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.424554    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls podName:b472852f-5f8d-4d93-9d0d-45fb4a24f564 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.424532549 +0000 UTC m=+39.255560059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9b7h5" (UID: "b472852f-5f8d-4d93-9d0d-45fb4a24f564") : secret "samples-operator-tls" not found
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.425030    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.425082    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls podName:915d900b-a1a6-428b-ac3a-0641215f0233 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.425066067 +0000 UTC m=+39.256093583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls") pod "dns-default-2jz8n" (UID: "915d900b-a1a6-428b-ac3a-0641215f0233") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.425136    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:05.425204 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.425169    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls podName:661b8c3f-b66f-4989-ae4d-2ceb5e3b831f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.425158306 +0000 UTC m=+39.256185818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n86rk" (UID: "661b8c3f-b66f-4989-ae4d-2ceb5e3b831f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:05.525830 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:05.525772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:05.526531 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:05.525836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:05.526531 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.526055    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:05.526531 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.526115    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.526097988 +0000 UTC m=+39.357125505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : secret "router-metrics-certs-default" not found
Sep 10 23:37:05.526531 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:05.526192    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.526180312 +0000 UTC m=+39.357207829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:06.943008 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:06.942779    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr" event={"ID":"a93fdd16-d399-4023-810e-d611f8f7aee0","Type":"ContainerStarted","Data":"922a31c3f2594607ca629a2b0ceb56ce66d0315e03fd97c75186a076e24a66e7"}
Sep 10 23:37:06.957791 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:06.957671    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r9bdr" podStartSLOduration=15.198335647 podStartE2EDuration="16.957653772s" totalImagesPullingTime="1.759318125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:06.957308448 +0000 UTC m=+38.788336005" watchObservedRunningTime="2026-09-10 23:37:06.957653772 +0000 UTC m=+38.788681306"
Sep 10 23:37:07.047430 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:07.047399    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:07.148872 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:07.148839    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:07.347168 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:07.347077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:07.347327 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.347253    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:07.347327 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.347274    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6c8fc5d7-jkp94: secret "image-registry-tls" not found
Sep 10 23:37:07.347440 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.347334    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls podName:576b260f-d773-4bd1-b98b-fdc0d99626d5 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.347313739 +0000 UTC m=+43.178341266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls") pod "image-registry-5c6c8fc5d7-jkp94" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5") : secret "image-registry-tls" not found
Sep 10 23:37:07.448279 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:07.448244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:07.448439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:07.448329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:07.448439 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.448361    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:07.448439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:07.448380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:07.448439 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.448425    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls podName:915d900b-a1a6-428b-ac3a-0641215f0233 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.448409082 +0000 UTC m=+43.279436597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls") pod "dns-default-2jz8n" (UID: "915d900b-a1a6-428b-ac3a-0641215f0233") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:07.448615 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.448486    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:07.448615 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.448490    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:07.448615 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.448542    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls podName:b472852f-5f8d-4d93-9d0d-45fb4a24f564 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.448525619 +0000 UTC m=+43.279553145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9b7h5" (UID: "b472852f-5f8d-4d93-9d0d-45fb4a24f564") : secret "samples-operator-tls" not found
Sep 10 23:37:07.448615 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.448559    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls podName:661b8c3f-b66f-4989-ae4d-2ceb5e3b831f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.448549519 +0000 UTC m=+43.279577032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n86rk" (UID: "661b8c3f-b66f-4989-ae4d-2ceb5e3b831f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:07.548945 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:07.548903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:07.548945 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:07.548952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:07.549181 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.549034    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:07.549181 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.549066    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.549043517 +0000 UTC m=+43.380071044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:07.549181 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:07.549092    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.549083068 +0000 UTC m=+43.380110584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : secret "router-metrics-certs-default" not found
Sep 10 23:37:08.476238 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.476213    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"]
Sep 10 23:37:08.496483 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.496461    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"]
Sep 10 23:37:08.496620 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.496576    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.499091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.499070    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 23:37:08.499091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.499089    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 23:37:08.499264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.499110    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 23:37:08.499264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.499070    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 23:37:08.510646 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.510627    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"]
Sep 10 23:37:08.510753 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.510740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.513129 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.513110    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 23:37:08.513295 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.513281    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 23:37:08.513352 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.513328    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 23:37:08.513447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.513431    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 23:37:08.539020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.538996    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"]
Sep 10 23:37:08.539113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.539034    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"]
Sep 10 23:37:08.539113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.539055    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"]
Sep 10 23:37:08.539113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.539105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"
Sep 10 23:37:08.541781 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.541760    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wdqmh\""
Sep 10 23:37:08.541781 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.541779    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 23:37:08.556206 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-hub\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.556297 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556257    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6nk\" (UniqueName: \"kubernetes.io/projected/fb13d7b5-b5ba-40b6-af79-f20db76cf277-kube-api-access-sf6nk\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.556338 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67d6d56868-czscw\" (UID: \"49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"
Sep 10 23:37:08.556444 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556427    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dftx8\" (UniqueName: \"kubernetes.io/projected/49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af-kube-api-access-dftx8\") pod \"managed-serviceaccount-addon-agent-67d6d56868-czscw\" (UID: \"49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"
Sep 10 23:37:08.556480 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.556517 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4rn\" (UniqueName: \"kubernetes.io/projected/ca1032cc-da23-4a72-a903-353801d363a5-kube-api-access-7h4rn\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.556550 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb13d7b5-b5ba-40b6-af79-f20db76cf277-additional-service-ca\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.556600 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556550    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.556649 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556633    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca1032cc-da23-4a72-a903-353801d363a5-klusterlet-config\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.556701 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca1032cc-da23-4a72-a903-353801d363a5-tmp\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.556749 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.556717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-ca\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.657250 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca1032cc-da23-4a72-a903-353801d363a5-klusterlet-config\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.657250 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca1032cc-da23-4a72-a903-353801d363a5-tmp\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.657474 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-ca\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.657474 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-hub\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.657474 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6nk\" (UniqueName: \"kubernetes.io/projected/fb13d7b5-b5ba-40b6-af79-f20db76cf277-kube-api-access-sf6nk\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.657474 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657392    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67d6d56868-czscw\" (UID: \"49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"
Sep 10 23:37:08.657670 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftx8\" (UniqueName: \"kubernetes.io/projected/49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af-kube-api-access-dftx8\") pod \"managed-serviceaccount-addon-agent-67d6d56868-czscw\" (UID: \"49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"
Sep 10 23:37:08.657670 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.657670 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4rn\" (UniqueName: \"kubernetes.io/projected/ca1032cc-da23-4a72-a903-353801d363a5-kube-api-access-7h4rn\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.657670 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657599    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb13d7b5-b5ba-40b6-af79-f20db76cf277-additional-service-ca\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.657670 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.657903 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.657719    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca1032cc-da23-4a72-a903-353801d363a5-tmp\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.658839 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.658809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb13d7b5-b5ba-40b6-af79-f20db76cf277-additional-service-ca\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.661169 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.661142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.661430 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.661385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67d6d56868-czscw\" (UID: \"49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"
Sep 10 23:37:08.662052 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.662024    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-ca\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.663337 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.662632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-hub\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.663337 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.662638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca1032cc-da23-4a72-a903-353801d363a5-klusterlet-config\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.663896 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.663870    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fb13d7b5-b5ba-40b6-af79-f20db76cf277-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.667491 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.667468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4rn\" (UniqueName: \"kubernetes.io/projected/ca1032cc-da23-4a72-a903-353801d363a5-kube-api-access-7h4rn\") pod \"klusterlet-addon-workmgr-64df4d7d74-pd8vl\" (UID: \"ca1032cc-da23-4a72-a903-353801d363a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.667647 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.667608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dftx8\" (UniqueName: \"kubernetes.io/projected/49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af-kube-api-access-dftx8\") pod \"managed-serviceaccount-addon-agent-67d6d56868-czscw\" (UID: \"49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"
Sep 10 23:37:08.668102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.668082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6nk\" (UniqueName: \"kubernetes.io/projected/fb13d7b5-b5ba-40b6-af79-f20db76cf277-kube-api-access-sf6nk\") pod \"cluster-proxy-proxy-agent-765895748b-q2nrb\" (UID: \"fb13d7b5-b5ba-40b6-af79-f20db76cf277\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.731587 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.731538    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:08.798860 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.798825    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:08.805593 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.805572    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:08.836614 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.836590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:08.846880 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:08.846854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"
Sep 10 23:37:09.083249 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:09.083222    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:09.184492 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:09.184471    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:10.308589 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:10.308558    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:10.421819 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:10.421777    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:10.477678 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:10.477653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:11.380648 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.380612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:11.381044 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.380761    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:11.381044 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.380779    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6c8fc5d7-jkp94: secret "image-registry-tls" not found
Sep 10 23:37:11.381044 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.380834    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls podName:576b260f-d773-4bd1-b98b-fdc0d99626d5 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.380817164 +0000 UTC m=+51.211844680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls") pod "image-registry-5c6c8fc5d7-jkp94" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5") : secret "image-registry-tls" not found
Sep 10 23:37:11.481905 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.481881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:11.482007 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.481980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:11.482065 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.482035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:11.482065 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.482059    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:11.482161 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.482114    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls podName:915d900b-a1a6-428b-ac3a-0641215f0233 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.482097842 +0000 UTC m=+51.313125355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls") pod "dns-default-2jz8n" (UID: "915d900b-a1a6-428b-ac3a-0641215f0233") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:11.482215 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.482167    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:11.482215 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.482174    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:11.482215 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.482205    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls podName:b472852f-5f8d-4d93-9d0d-45fb4a24f564 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.482194937 +0000 UTC m=+51.313222462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9b7h5" (UID: "b472852f-5f8d-4d93-9d0d-45fb4a24f564") : secret "samples-operator-tls" not found
Sep 10 23:37:11.482338 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.482243    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls podName:661b8c3f-b66f-4989-ae4d-2ceb5e3b831f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.482217824 +0000 UTC m=+51.313245354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n86rk" (UID: "661b8c3f-b66f-4989-ae4d-2ceb5e3b831f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:11.583344 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.583321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:11.583430 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.583360    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:11.583493 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.583480    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:11.583524 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.583493    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.583476382 +0000 UTC m=+51.414503910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:11.583524 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:11.583520    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.583510716 +0000 UTC m=+51.414538226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : secret "router-metrics-certs-default" not found
Sep 10 23:37:11.720505 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.720415    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:11.811458 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.811432    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:11.952800 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.952740    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw"]
Sep 10 23:37:11.958194 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.958154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"]
Sep 10 23:37:11.962759 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.962415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5" event={"ID":"4ab50fa8-0838-473c-85e9-11dfd2ebdc35","Type":"ContainerStarted","Data":"555903b0d5cbbc045f449da64256df589997dfcb95ada4605cdd5b8da4552833"}
Sep 10 23:37:11.963073 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.963053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"]
Sep 10 23:37:11.966059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.965595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jvt2x" event={"ID":"156442c4-f848-4a58-899f-26ea730df356","Type":"ContainerStarted","Data":"3addada45b26a5c574ccc6b2ea97ee07152c6c6d2581569d63e079d9e458169f"}
Sep 10 23:37:11.968053 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.967997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw" event={"ID":"96674fff-8032-41fb-b75e-ca10f7d60860","Type":"ContainerStarted","Data":"ddb1d4251c8c96469de9f547c87cfc85f8493e356b9fb42ef26d3e7a89c7b002"}
Sep 10 23:37:11.970384 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.969914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wdgcw" event={"ID":"17c84d6c-7d15-4b25-be27-78c331ceb2ae","Type":"ContainerStarted","Data":"fd07c9c8550851d4e5218d7c3d5f6f71ce06f9b34388dfc39feebac547290bf2"}
Sep 10 23:37:11.972107 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.972089    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/0.log"
Sep 10 23:37:11.972214 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.972122    2562 generic.go:356] "Generic (PLEG): container finished" podID="43537109-e795-4d12-9269-b5578783dc5e" containerID="e1170bf25d0b9f3d40314f4152df2f9f9e50f8e0309a4aba01341cdf0128cb2e" exitCode=255
Sep 10 23:37:11.972412 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.972360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf" event={"ID":"43537109-e795-4d12-9269-b5578783dc5e","Type":"ContainerDied","Data":"e1170bf25d0b9f3d40314f4152df2f9f9e50f8e0309a4aba01341cdf0128cb2e"}
Sep 10 23:37:11.972498 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.972434    2562 scope.go:117] "RemoveContainer" containerID="e1170bf25d0b9f3d40314f4152df2f9f9e50f8e0309a4aba01341cdf0128cb2e"
Sep 10 23:37:11.975382 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.974535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s" event={"ID":"356212ea-1e0e-452a-8e4c-a270b6c46ff7","Type":"ContainerStarted","Data":"348b329243a298be91b2537937ef0cef2a3524d3dd97675121f705f785029918"}
Sep 10 23:37:11.981421 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:11.981369    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mdqv5" podStartSLOduration=14.680820847 podStartE2EDuration="21.981353127s" totalImagesPullingTime="7.30053228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:11.979859788 +0000 UTC m=+43.810887322" watchObservedRunningTime="2026-09-10 23:37:11.981353127 +0000 UTC m=+43.812380662"
Sep 10 23:37:11.984253 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:11.984217    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb13d7b5_b5ba_40b6_af79_f20db76cf277.slice/crio-5ed33330a3051d6db951d7975132af8ce5a677acd2b49b61319244084867dda6 WatchSource:0}: Error finding container 5ed33330a3051d6db951d7975132af8ce5a677acd2b49b61319244084867dda6: Status 404 returned error can't find the container with id 5ed33330a3051d6db951d7975132af8ce5a677acd2b49b61319244084867dda6
Sep 10 23:37:12.009628 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:12.009586    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jvt2x" podStartSLOduration=1.716947367 podStartE2EDuration="9.009557551s" totalImagesPullingTime="7.292610184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:12.009135777 +0000 UTC m=+43.840163335" watchObservedRunningTime="2026-09-10 23:37:12.009557551 +0000 UTC m=+43.840585085"
Sep 10 23:37:12.046771 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:12.046590    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw" podStartSLOduration=14.596864523 podStartE2EDuration="22.046571626s" totalImagesPullingTime="7.449707103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:12.046330677 +0000 UTC m=+43.877358206" watchObservedRunningTime="2026-09-10 23:37:12.046571626 +0000 UTC m=+43.877599160"
Sep 10 23:37:12.067417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:12.067374    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-wdgcw" podStartSLOduration=14.65901822 podStartE2EDuration="22.06735959s" totalImagesPullingTime="7.40834137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:12.06651495 +0000 UTC m=+43.897542486" watchObservedRunningTime="2026-09-10 23:37:12.06735959 +0000 UTC m=+43.898387136"
Sep 10 23:37:12.370239 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:12.370201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:12.468017 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:12.467985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:12.990488 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:12.990374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl" event={"ID":"ca1032cc-da23-4a72-a903-353801d363a5","Type":"ContainerStarted","Data":"f2fe9d9f18b3eee10d783c4bd091414fbb0685be83a8d8caa90ff216a6226ae5"}
Sep 10 23:37:12.990662 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:12.990607    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:12.997451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw" event={"ID":"49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af","Type":"ContainerStarted","Data":"2f1f2da2180fabc019a024ead0e0b9b35bdcf8f7def181322c075a7f56fa3564"}
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.000801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" event={"ID":"fb13d7b5-b5ba-40b6-af79-f20db76cf277","Type":"ContainerStarted","Data":"5ed33330a3051d6db951d7975132af8ce5a677acd2b49b61319244084867dda6"}
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.006862    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.007571    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/0.log"
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.007603    2562 generic.go:356] "Generic (PLEG): container finished" podID="43537109-e795-4d12-9269-b5578783dc5e" containerID="0bd887ada3dd7ff7f1585d5fe6794264a8d37767ea45d56800dd4d13458c60df" exitCode=255
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.008334    2562 scope.go:117] "RemoveContainer" containerID="0bd887ada3dd7ff7f1585d5fe6794264a8d37767ea45d56800dd4d13458c60df"
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:13.008490    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-gw9pf_openshift-console-operator(43537109-e795-4d12-9269-b5578783dc5e)\"" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf" podUID="43537109-e795-4d12-9269-b5578783dc5e"
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.008714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf" event={"ID":"43537109-e795-4d12-9269-b5578783dc5e","Type":"ContainerDied","Data":"0bd887ada3dd7ff7f1585d5fe6794264a8d37767ea45d56800dd4d13458c60df"}
Sep 10 23:37:13.010273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.008764    2562 scope.go:117] "RemoveContainer" containerID="e1170bf25d0b9f3d40314f4152df2f9f9e50f8e0309a4aba01341cdf0128cb2e"
Sep 10 23:37:13.061828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.061777    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s" podStartSLOduration=15.701762141 podStartE2EDuration="23.061758415s" totalImagesPullingTime="7.359996274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:12.081232449 +0000 UTC m=+43.912260009" watchObservedRunningTime="2026-09-10 23:37:13.061758415 +0000 UTC m=+44.892785945"
Sep 10 23:37:13.084799 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.084744    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:13.698431 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.698399    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:13.795820 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.795785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:13.901113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.900307    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v9r7s"]
Sep 10 23:37:13.928807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.928712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:13.940760 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.940734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:13.940903 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.940780    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v9r7s"]
Sep 10 23:37:13.940903 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.940843    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:13.941113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.940959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:13.944447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.944006    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 23:37:13.944447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.944233    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bptlm\""
Sep 10 23:37:13.944812 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.944589    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 23:37:13.998896 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:13.998844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.015893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.015569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.015893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.015608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.015893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.015668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13e04c4f-4b29-4167-af11-0f11b8810fae-crio-socket\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.015893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.015724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwl9d\" (UniqueName: \"kubernetes.io/projected/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-api-access-wwl9d\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.015893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.015754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13e04c4f-4b29-4167-af11-0f11b8810fae-data-volume\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.016635 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.016616    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:37:14.017073 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.017020    2562 scope.go:117] "RemoveContainer" containerID="0bd887ada3dd7ff7f1585d5fe6794264a8d37767ea45d56800dd4d13458c60df"
Sep 10 23:37:14.017226 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.017204    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-gw9pf_openshift-console-operator(43537109-e795-4d12-9269-b5578783dc5e)\"" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf" podUID="43537109-e795-4d12-9269-b5578783dc5e"
Sep 10 23:37:14.119138 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.119096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.119300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.119151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.119300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.119233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13e04c4f-4b29-4167-af11-0f11b8810fae-crio-socket\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.119300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.119295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwl9d\" (UniqueName: \"kubernetes.io/projected/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-api-access-wwl9d\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.119460 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.119324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13e04c4f-4b29-4167-af11-0f11b8810fae-data-volume\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.119460 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.119382    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.119460 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.119444    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.619425383 +0000 UTC m=+46.450452904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.119623 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.119533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13e04c4f-4b29-4167-af11-0f11b8810fae-crio-socket\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.119623 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.119612    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.119723 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.119662    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.619645013 +0000 UTC m=+46.450672538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.120251 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.120197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13e04c4f-4b29-4167-af11-0f11b8810fae-data-volume\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.144669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.144594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwl9d\" (UniqueName: \"kubernetes.io/projected/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-api-access-wwl9d\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.388461 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.388278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.494482 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.494377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.628271 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.627502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.628271 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.627547    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:14.628271 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.627710    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.628271 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.627772    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.627753416 +0000 UTC m=+47.458780935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.628271 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.628192    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.628271 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:14.628244    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.628229264 +0000 UTC m=+47.459256777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.694448 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.694412    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:14.766349 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:14.766317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:15.015766 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:15.015675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:15.020725 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:15.020701    2562 scope.go:117] "RemoveContainer" containerID="0bd887ada3dd7ff7f1585d5fe6794264a8d37767ea45d56800dd4d13458c60df"
Sep 10 23:37:15.021025 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:15.020913    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-gw9pf_openshift-console-operator(43537109-e795-4d12-9269-b5578783dc5e)\"" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf" podUID="43537109-e795-4d12-9269-b5578783dc5e"
Sep 10 23:37:15.123570 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:15.123537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:15.365137 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:15.365014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:15.434489 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:15.434388    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:15.467454 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:15.467420    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gvg49_1e324660-2423-4acb-becf-9a78be67198a/dns-node-resolver/0.log"
Sep 10 23:37:15.638700 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:15.638602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:15.638700 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:15.638647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:15.639066 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:15.638735    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:15.639066 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:15.638736    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:15.639066 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:15.638799    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:17.638779393 +0000 UTC m=+49.469806934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:15.639066 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:15.638829    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:17.638814482 +0000 UTC m=+49.469841993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:16.150091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:16.150057    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:16.270365 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:16.270338    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2q2tj_6f736cd6-ad74-4550-a2a8-82f59ee9d98f/node-ca/0.log"
Sep 10 23:37:17.272023 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:17.271993    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jvt2x_156442c4-f848-4a58-899f-26ea730df356/serve-healthcheck-canary/0.log"
Sep 10 23:37:17.659654 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:17.659571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:17.659654 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:17.659618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:17.659858 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:17.659738    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:17.659858 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:17.659782    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:17.659858 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:17.659823    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:21.659800696 +0000 UTC m=+53.490828224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:17.659858 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:17.659846    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:21.659833991 +0000 UTC m=+53.490861503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:19.417079 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.417047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.477507 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.477462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:19.477679 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.477631    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:19.477679 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.477653    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6c8fc5d7-jkp94: secret "image-registry-tls" not found
Sep 10 23:37:19.477771 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.477725    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls podName:576b260f-d773-4bd1-b98b-fdc0d99626d5 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:35.477710687 +0000 UTC m=+67.308738203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls") pod "image-registry-5c6c8fc5d7-jkp94" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5") : secret "image-registry-tls" not found
Sep 10 23:37:19.518044 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.517997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.578519 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.578481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:19.578656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.578578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:19.578656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.578631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:19.578656 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.578636    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:19.578840 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.578678    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:19.578840 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.578695    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls podName:915d900b-a1a6-428b-ac3a-0641215f0233 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:35.578675506 +0000 UTC m=+67.409703023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls") pod "dns-default-2jz8n" (UID: "915d900b-a1a6-428b-ac3a-0641215f0233") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:19.578840 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.578741    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls podName:661b8c3f-b66f-4989-ae4d-2ceb5e3b831f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:35.578723873 +0000 UTC m=+67.409751398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n86rk" (UID: "661b8c3f-b66f-4989-ae4d-2ceb5e3b831f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:37:19.578840 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.578746    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:37:19.578840 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.578784    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls podName:b472852f-5f8d-4d93-9d0d-45fb4a24f564 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:35.578772876 +0000 UTC m=+67.409800407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9b7h5" (UID: "b472852f-5f8d-4d93-9d0d-45fb4a24f564") : secret "samples-operator-tls" not found
Sep 10 23:37:19.590192 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.590158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.660240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.660206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:19.680177 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.680121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:19.680177 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.680150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:19.680313 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.680257    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:35.680240447 +0000 UTC m=+67.511267962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : configmap references non-existent config key: service-ca.crt
Sep 10 23:37:19.680313 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.680301    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:37:19.680388 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:19.680349    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs podName:12bf9c01-3b14-47e8-97a1-cd867914835f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:35.680336957 +0000 UTC m=+67.511364468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs") pod "router-default-5c5d97497d-trwfb" (UID: "12bf9c01-3b14-47e8-97a1-cd867914835f") : secret "router-metrics-certs-default" not found
Sep 10 23:37:19.759441 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:19.759419    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:20.300161 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:20.300132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:20.698883 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:20.698851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:21.041378 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.041293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl" event={"ID":"ca1032cc-da23-4a72-a903-353801d363a5","Type":"ContainerStarted","Data":"1871ea555a4506305267b972ac48d4354304342c912cd94a89410fe82e90958f"}
Sep 10 23:37:21.041378 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.041350    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:21.042944 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.042908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw" event={"ID":"49ad0fe9-fda9-4d33-8232-3ed4ab7ea4af","Type":"ContainerStarted","Data":"b7365c160000401f65c088940b8ecad93729c1d8e7133854f22189656bb00d4b"}
Sep 10 23:37:21.043367 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.043346    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl"
Sep 10 23:37:21.044817 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.044795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" event={"ID":"fb13d7b5-b5ba-40b6-af79-f20db76cf277","Type":"ContainerStarted","Data":"a05b31423310c85316527addb0ba405531a55cad55240becff68a82350d9df7b"}
Sep 10 23:37:21.044898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.044828    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" event={"ID":"fb13d7b5-b5ba-40b6-af79-f20db76cf277","Type":"ContainerStarted","Data":"718e585435887ca8393fd9c93b24b0bd777ff4cd7f5d01a2d17b0c1a4e7c1044"}
Sep 10 23:37:21.044898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.044846    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" event={"ID":"fb13d7b5-b5ba-40b6-af79-f20db76cf277","Type":"ContainerStarted","Data":"edcd21331587506ba45cd2c42f72f1dc09b62a4968254af564631ce552345666"}
Sep 10 23:37:21.060308 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.060268    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64df4d7d74-pd8vl" podStartSLOduration=4.977435277 podStartE2EDuration="13.060255326s" totalImagesPullingTime="8.082820049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:21.059341522 +0000 UTC m=+52.890369090" watchObservedRunningTime="2026-09-10 23:37:21.060255326 +0000 UTC m=+52.891282858"
Sep 10 23:37:21.075372 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.075337    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d6d56868-czscw" podStartSLOduration=5.011714426 podStartE2EDuration="13.075325241s" totalImagesPullingTime="8.063610815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:21.074962051 +0000 UTC m=+52.905989580" watchObservedRunningTime="2026-09-10 23:37:21.075325241 +0000 UTC m=+52.906352773"
Sep 10 23:37:21.110072 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.110021    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" podStartSLOduration=5.039799669 podStartE2EDuration="13.110009167s" totalImagesPullingTime="8.070209498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:21.108802896 +0000 UTC m=+52.939830562" watchObservedRunningTime="2026-09-10 23:37:21.110009167 +0000 UTC m=+52.941036699"
Sep 10 23:37:21.306342 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.306278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:21.698637 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.698601    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:21.698637 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.698642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:21.698823 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:21.698745    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:21.698823 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:21.698799    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls podName:13e04c4f-4b29-4167-af11-0f11b8810fae nodeName:}" failed. No retries permitted until 2026-09-10 23:37:29.698784627 +0000 UTC m=+61.529812137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9r7s" (UID: "13e04c4f-4b29-4167-af11-0f11b8810fae") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:21.699201 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.699185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13e04c4f-4b29-4167-af11-0f11b8810fae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:21.904321 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:21.904297    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:22.106599 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:22.106530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:22.304393 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:22.304365    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:22.415090 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:22.415001    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:22.515716 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:22.515689    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:22.570851 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:22.570828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:22.626453 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:22.626429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.016251 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:23.016224    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.111874 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:23.111854    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.213317 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:23.213297    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.310956 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:23.310883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.502058 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:23.502030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.704662 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:23.704637    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:23.759297 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:23.759276    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:24.113948 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:24.113863    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:24.301820 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:24.301796    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:24.698100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:24.698072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:25.302133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:25.302102    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:25.685144 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:25.685114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:25.902425 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:25.902389    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:26.100915 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:26.100858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:26.300427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:26.300405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:26.502174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:26.502147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:27.200467 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:27.200439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:27.253872 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:27.253845    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:27.707060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:27.707032    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:27.900296 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:27.900270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:28.107639 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:28.107554    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:28.299909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:28.299881    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:28.510964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:28.510921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:28.839081 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:28.838993    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" podUID="fb13d7b5-b5ba-40b6-af79-f20db76cf277" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 23:37:29.111466 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.111391    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:29.649377 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.649353    2562 scope.go:117] "RemoveContainer" containerID="0bd887ada3dd7ff7f1585d5fe6794264a8d37767ea45d56800dd4d13458c60df"
Sep 10 23:37:29.707247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.707223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:29.759444 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.759417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:29.762165 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.762142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13e04c4f-4b29-4167-af11-0f11b8810fae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9r7s\" (UID: \"13e04c4f-4b29-4167-af11-0f11b8810fae\") " pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:29.860774 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.860747    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bptlm\""
Sep 10 23:37:29.868637 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.868616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9r7s"
Sep 10 23:37:29.901099 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.901039    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sj7"
Sep 10 23:37:29.905575 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:29.905537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:30.072685 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.072662    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:37:30.072828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.072748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf" event={"ID":"43537109-e795-4d12-9269-b5578783dc5e","Type":"ContainerStarted","Data":"5cd878cee35c7056a812d5360e36a213652c6a55f1b594cc577ec57f00d6d5cb"}
Sep 10 23:37:30.073056 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.073034    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:30.078349 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.078326    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf"
Sep 10 23:37:30.091011 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.090960    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-gw9pf" podStartSLOduration=32.642429484 podStartE2EDuration="40.090922346s" totalImagesPullingTime="7.448492862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:30.090450938 +0000 UTC m=+61.921478495" watchObservedRunningTime="2026-09-10 23:37:30.090922346 +0000 UTC m=+61.921949882"
Sep 10 23:37:30.118513 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.118466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:30.158001 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.157920    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v9r7s"]
Sep 10 23:37:30.160958 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:30.160913    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e04c4f_4b29_4167_af11_0f11b8810fae.slice/crio-bc5b0ba2b8098b8b93fa0d5879581e0581d1a25792d56b65f66ac441f823b4f2 WatchSource:0}: Error finding container bc5b0ba2b8098b8b93fa0d5879581e0581d1a25792d56b65f66ac441f823b4f2: Status 404 returned error can't find the container with id bc5b0ba2b8098b8b93fa0d5879581e0581d1a25792d56b65f66ac441f823b4f2
Sep 10 23:37:30.244289 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.244263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:30.347501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.347471    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:30.445383 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.445360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:30.546810 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.546788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:30.927530 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:30.927448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:31.028048 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.028009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:31.080841 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.080798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9r7s" event={"ID":"13e04c4f-4b29-4167-af11-0f11b8810fae","Type":"ContainerStarted","Data":"a40db56a123616648af117a8ca9d4b16d6c3a07b10142a576ba089c00796a79d"}
Sep 10 23:37:31.081038 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.080904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9r7s" event={"ID":"13e04c4f-4b29-4167-af11-0f11b8810fae","Type":"ContainerStarted","Data":"bc5b0ba2b8098b8b93fa0d5879581e0581d1a25792d56b65f66ac441f823b4f2"}
Sep 10 23:37:31.332208 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.332170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:31.550264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.550235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:31.652021 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.651995    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:31.752472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.752448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:31.806869 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.806842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:31.937138 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:31.937115    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:32.086655 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:32.086626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9r7s" event={"ID":"13e04c4f-4b29-4167-af11-0f11b8810fae","Type":"ContainerStarted","Data":"402378921e67fccf7006c74fd8fade25dbbc40f1484edf0c7e0557aeaf482f1f"}
Sep 10 23:37:32.104339 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:32.104313    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:33.950436 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:33.950401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:34.004263 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.004233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:34.094779 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.094748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9r7s" event={"ID":"13e04c4f-4b29-4167-af11-0f11b8810fae","Type":"ContainerStarted","Data":"a90ca8be41403a05383d2b4d5194ba0afb587b344eeee67c16b70ad62bba7f74"}
Sep 10 23:37:34.114373 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.114333    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v9r7s" podStartSLOduration=18.143442718 podStartE2EDuration="21.114319389s" totalImagesPullingTime="2.970876671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:34.113636979 +0000 UTC m=+65.944664537" watchObservedRunningTime="2026-09-10 23:37:34.114319389 +0000 UTC m=+65.945346922"
Sep 10 23:37:34.402584 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.402523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:37:34.402584 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.402559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:37:34.405576 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.405555    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:37:34.405802 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.405785    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:37:34.415835 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.415814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c15c11b-b1b2-4c3b-a4c8-edc70777af3d-original-pull-secret\") pod \"global-pull-secret-syncer-qmbgq\" (UID: \"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d\") " pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:37:34.415971 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.415955    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745744e6-2b42-4e82-bfba-6c2f79f57633-metrics-certs\") pod \"network-metrics-daemon-m8gj9\" (UID: \"745744e6-2b42-4e82-bfba-6c2f79f57633\") " pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:37:34.503577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.503550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:37:34.505708 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.505692    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94c8\" (UniqueName: \"kubernetes.io/projected/333a5c94-32b1-4b76-9e90-39206904ddce-kube-api-access-x94c8\") pod \"network-check-target-27vsj\" (UID: \"333a5c94-32b1-4b76-9e90-39206904ddce\") " pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:37:34.664961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.664888    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n6qg4\""
Sep 10 23:37:34.672987 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.672967    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8gj9"
Sep 10 23:37:34.674304 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.674286    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ppbqc\""
Sep 10 23:37:34.678160 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.678144    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbgq"
Sep 10 23:37:34.682277 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.682263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:37:34.975663 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.975638    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qmbgq"]
Sep 10 23:37:34.977988 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.977963    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m8gj9"]
Sep 10 23:37:34.978985 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:34.978961    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c15c11b_b1b2_4c3b_a4c8_edc70777af3d.slice/crio-f78c970d68d518a373d485b7dd0f5c9340e732d7ba3832f5189d24ed140c6952 WatchSource:0}: Error finding container f78c970d68d518a373d485b7dd0f5c9340e732d7ba3832f5189d24ed140c6952: Status 404 returned error can't find the container with id f78c970d68d518a373d485b7dd0f5c9340e732d7ba3832f5189d24ed140c6952
Sep 10 23:37:34.980122 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:34.980103    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745744e6_2b42_4e82_bfba_6c2f79f57633.slice/crio-5f64d94c9774c5e3e7b675426df78d9f822b38b386ec32eaede5d3d223b44bb7 WatchSource:0}: Error finding container 5f64d94c9774c5e3e7b675426df78d9f822b38b386ec32eaede5d3d223b44bb7: Status 404 returned error can't find the container with id 5f64d94c9774c5e3e7b675426df78d9f822b38b386ec32eaede5d3d223b44bb7
Sep 10 23:37:34.988725 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:34.988701    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-27vsj"]
Sep 10 23:37:34.990836 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:34.990815    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333a5c94_32b1_4b76_9e90_39206904ddce.slice/crio-527b492474315753dbf0fd3cb3a0ffa8e3b2c420b54588314d23e3109e6d318b WatchSource:0}: Error finding container 527b492474315753dbf0fd3cb3a0ffa8e3b2c420b54588314d23e3109e6d318b: Status 404 returned error can't find the container with id 527b492474315753dbf0fd3cb3a0ffa8e3b2c420b54588314d23e3109e6d318b
Sep 10 23:37:35.099185 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.099155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-27vsj" event={"ID":"333a5c94-32b1-4b76-9e90-39206904ddce","Type":"ContainerStarted","Data":"d508aa0366e86412b086c44b550c42ab0a0b2220795d2f39c8697d2edca2f8b2"}
Sep 10 23:37:35.099295 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.099249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-27vsj" event={"ID":"333a5c94-32b1-4b76-9e90-39206904ddce","Type":"ContainerStarted","Data":"527b492474315753dbf0fd3cb3a0ffa8e3b2c420b54588314d23e3109e6d318b"}
Sep 10 23:37:35.099295 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.099279    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:37:35.100439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.100416    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qmbgq" event={"ID":"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d","Type":"ContainerStarted","Data":"f78c970d68d518a373d485b7dd0f5c9340e732d7ba3832f5189d24ed140c6952"}
Sep 10 23:37:35.101539 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.101518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8gj9" event={"ID":"745744e6-2b42-4e82-bfba-6c2f79f57633","Type":"ContainerStarted","Data":"5f64d94c9774c5e3e7b675426df78d9f822b38b386ec32eaede5d3d223b44bb7"}
Sep 10 23:37:35.127318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.127276    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-27vsj" podStartSLOduration=66.12726371 podStartE2EDuration="1m6.12726371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:35.127029531 +0000 UTC m=+66.958057085" watchObservedRunningTime="2026-09-10 23:37:35.12726371 +0000 UTC m=+66.958291242"
Sep 10 23:37:35.461102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.461066    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:35.515871 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.515834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:35.518908 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.518880    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"image-registry-5c6c8fc5d7-jkp94\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") " pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:35.617275 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.617240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:35.617442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.617331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:35.617442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.617372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:35.619984 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.619959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/915d900b-a1a6-428b-ac3a-0641215f0233-metrics-tls\") pod \"dns-default-2jz8n\" (UID: \"915d900b-a1a6-428b-ac3a-0641215f0233\") " pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:35.621893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.621868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661b8c3f-b66f-4989-ae4d-2ceb5e3b831f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n86rk\" (UID: \"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:35.622030 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.622006    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b472852f-5f8d-4d93-9d0d-45fb4a24f564-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9b7h5\" (UID: \"b472852f-5f8d-4d93-9d0d-45fb4a24f564\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:35.646837 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.646774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:35.656909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.656888    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wsf9n\""
Sep 10 23:37:35.665002 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.664981    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:35.718421 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.718346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:35.718421 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.718393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:35.719165 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.719140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12bf9c01-3b14-47e8-97a1-cd867914835f-service-ca-bundle\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:35.721339 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.721317    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bf9c01-3b14-47e8-97a1-cd867914835f-metrics-certs\") pod \"router-default-5c5d97497d-trwfb\" (UID: \"12bf9c01-3b14-47e8-97a1-cd867914835f\") " pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:35.765342 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.765129    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8n4tg\""
Sep 10 23:37:35.770107 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.770050    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"
Sep 10 23:37:35.774986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.774961    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:35.775949 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.775916    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9sdk\""
Sep 10 23:37:35.784020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.783990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:35.811581 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.811553    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7gx8t\""
Sep 10 23:37:35.815349 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.815299    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"
Sep 10 23:37:35.846110 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.846087    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4hzqj\""
Sep 10 23:37:35.853770 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.853748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:35.893112 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:35.893084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:36.281735 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.281539    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk"]
Sep 10 23:37:36.282228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.282167    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2jz8n"]
Sep 10 23:37:36.282709 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:36.282674    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915d900b_a1a6_428b_ac3a_0641215f0233.slice/crio-7d97a41a9f4aa78d602605f7e9fc6a860e032a5de68d4cb0b81a4f585f5b0e92 WatchSource:0}: Error finding container 7d97a41a9f4aa78d602605f7e9fc6a860e032a5de68d4cb0b81a4f585f5b0e92: Status 404 returned error can't find the container with id 7d97a41a9f4aa78d602605f7e9fc6a860e032a5de68d4cb0b81a4f585f5b0e92
Sep 10 23:37:36.282976 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:36.282953    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661b8c3f_b66f_4989_ae4d_2ceb5e3b831f.slice/crio-eb7b6b6801662f87ceebb3e0b774991dee6fe77c52b19dfa0618702a083f9a7a WatchSource:0}: Error finding container eb7b6b6801662f87ceebb3e0b774991dee6fe77c52b19dfa0618702a083f9a7a: Status 404 returned error can't find the container with id eb7b6b6801662f87ceebb3e0b774991dee6fe77c52b19dfa0618702a083f9a7a
Sep 10 23:37:36.315455 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.315419    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5"]
Sep 10 23:37:36.316837 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.316811    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"]
Sep 10 23:37:36.320213 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:36.320181    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576b260f_d773_4bd1_b98b_fdc0d99626d5.slice/crio-72d3ea5b5f2f1c8637b548f15a8f3de146f0c47b638285d559833eb29f499106 WatchSource:0}: Error finding container 72d3ea5b5f2f1c8637b548f15a8f3de146f0c47b638285d559833eb29f499106: Status 404 returned error can't find the container with id 72d3ea5b5f2f1c8637b548f15a8f3de146f0c47b638285d559833eb29f499106
Sep 10 23:37:36.322464 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.322407    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c5d97497d-trwfb"]
Sep 10 23:37:36.325787 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:36.325763    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bf9c01_3b14_47e8_97a1_cd867914835f.slice/crio-c64f525c3b46ddfa6ab566663faf5a210eab98c9dc09dc566caf115142b0c7a5 WatchSource:0}: Error finding container c64f525c3b46ddfa6ab566663faf5a210eab98c9dc09dc566caf115142b0c7a5: Status 404 returned error can't find the container with id c64f525c3b46ddfa6ab566663faf5a210eab98c9dc09dc566caf115142b0c7a5
Sep 10 23:37:36.349309 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.349280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:36.448310 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.448272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:36.724659 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.724616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:36.797169 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:36.797086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:37.116486 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.116407    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk" event={"ID":"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f","Type":"ContainerStarted","Data":"eb7b6b6801662f87ceebb3e0b774991dee6fe77c52b19dfa0618702a083f9a7a"}
Sep 10 23:37:37.120086 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.119572    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8gj9" event={"ID":"745744e6-2b42-4e82-bfba-6c2f79f57633","Type":"ContainerStarted","Data":"937c5072d4a56725caa6be3d0d4d4fde1b37b7955329b1b84b087a27340a7267"}
Sep 10 23:37:37.120086 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.119622    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8gj9" event={"ID":"745744e6-2b42-4e82-bfba-6c2f79f57633","Type":"ContainerStarted","Data":"aba76bda2e62fd7625bb595c4b0cccf4dada5b7a560ad74ebdbe656c5198ef5c"}
Sep 10 23:37:37.122964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.122574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94" event={"ID":"576b260f-d773-4bd1-b98b-fdc0d99626d5","Type":"ContainerStarted","Data":"281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc"}
Sep 10 23:37:37.122964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.122620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94" event={"ID":"576b260f-d773-4bd1-b98b-fdc0d99626d5","Type":"ContainerStarted","Data":"72d3ea5b5f2f1c8637b548f15a8f3de146f0c47b638285d559833eb29f499106"}
Sep 10 23:37:37.122964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.122663    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:37.125625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.125564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2jz8n" event={"ID":"915d900b-a1a6-428b-ac3a-0641215f0233","Type":"ContainerStarted","Data":"7d97a41a9f4aa78d602605f7e9fc6a860e032a5de68d4cb0b81a4f585f5b0e92"}
Sep 10 23:37:37.127581 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.127529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5" event={"ID":"b472852f-5f8d-4d93-9d0d-45fb4a24f564","Type":"ContainerStarted","Data":"8bee5e0ffb60bc693445f94e3acdced0dff5ef847b605461054b5a6969ce5ff1"}
Sep 10 23:37:37.131319 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.131298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c5d97497d-trwfb" event={"ID":"12bf9c01-3b14-47e8-97a1-cd867914835f","Type":"ContainerStarted","Data":"81192f00f5051be98844bcecfc5a5ec7e78e8b09eefec9750ac487b6e097a71a"}
Sep 10 23:37:37.131438 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.131335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c5d97497d-trwfb" event={"ID":"12bf9c01-3b14-47e8-97a1-cd867914835f","Type":"ContainerStarted","Data":"c64f525c3b46ddfa6ab566663faf5a210eab98c9dc09dc566caf115142b0c7a5"}
Sep 10 23:37:37.147686 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.147639    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m8gj9" podStartSLOduration=66.919269595 podStartE2EDuration="1m8.147621674s" totalImagesPullingTime="1.228352079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:37.1460995 +0000 UTC m=+68.977127052" watchObservedRunningTime="2026-09-10 23:37:37.147621674 +0000 UTC m=+68.978649209"
Sep 10 23:37:37.170480 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.170436    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94" podStartSLOduration=68.170420024 podStartE2EDuration="1m8.170420024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:37.169656558 +0000 UTC m=+69.000684113" watchObservedRunningTime="2026-09-10 23:37:37.170420024 +0000 UTC m=+69.001447557"
Sep 10 23:37:37.191539 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.191495    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c5d97497d-trwfb" podStartSLOduration=47.191479199 podStartE2EDuration="47.191479199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:37.190465672 +0000 UTC m=+69.021493239" watchObservedRunningTime="2026-09-10 23:37:37.191479199 +0000 UTC m=+69.022506733"
Sep 10 23:37:37.195315 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.195288    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:37.268710 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.268677    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:37.530858 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.530784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:37.854162 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.854087    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:37.856879 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.856855    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:37.944815 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:37.944780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:38.019571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:38.019535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:38.134998 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:38.134913    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:38.136123 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:38.136102    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c5d97497d-trwfb"
Sep 10 23:37:38.837679 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:38.837639    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" podUID="fb13d7b5-b5ba-40b6-af79-f20db76cf277" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 23:37:39.080742 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:39.080700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:39.139525 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:39.139442    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:42.093888 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.093852    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:42.154753 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.154666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk" event={"ID":"661b8c3f-b66f-4989-ae4d-2ceb5e3b831f","Type":"ContainerStarted","Data":"60f11a4620f681a0081bd33d359ea37d02357534d7b94e0c7c7e6f3a63f6bd18"}
Sep 10 23:37:42.156532 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.156507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2jz8n" event={"ID":"915d900b-a1a6-428b-ac3a-0641215f0233","Type":"ContainerStarted","Data":"42e92963d4f570ea48552ca7c829b3ea0bdacaca2c12560d9026761daf2c3352"}
Sep 10 23:37:42.156662 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.156550    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2jz8n" event={"ID":"915d900b-a1a6-428b-ac3a-0641215f0233","Type":"ContainerStarted","Data":"4264716b0820c16f8b0fbbd1223861ffb3b189b1af50bf05fab43eb2ccaccc75"}
Sep 10 23:37:42.156662 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.156578    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:42.158659 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.158633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5" event={"ID":"b472852f-5f8d-4d93-9d0d-45fb4a24f564","Type":"ContainerStarted","Data":"9221ddf1c46145ef4e01133fd3b114b8a1e0e7cfb85df4d466bd0976a8d101cb"}
Sep 10 23:37:42.158750 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.158686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5" event={"ID":"b472852f-5f8d-4d93-9d0d-45fb4a24f564","Type":"ContainerStarted","Data":"ff00daef8d5afe61ced9acb38eac34c03e9639474b91499906f663ad407407b4"}
Sep 10 23:37:42.160039 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.160016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qmbgq" event={"ID":"3c15c11b-b1b2-4c3b-a4c8-edc70777af3d","Type":"ContainerStarted","Data":"f822983c64952b690bea12aae5120bac9c0f77cc8ab7276033ad36f2a8b5a1e2"}
Sep 10 23:37:42.172828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.172787    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n86rk" podStartSLOduration=46.859934369 podStartE2EDuration="52.172775732s" totalImagesPullingTime="5.312841363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:42.172655993 +0000 UTC m=+74.003683548" watchObservedRunningTime="2026-09-10 23:37:42.172775732 +0000 UTC m=+74.003803264"
Sep 10 23:37:42.174651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.174631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:42.194496 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.194455    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qmbgq" podStartSLOduration=66.562992938 podStartE2EDuration="1m13.19444194s" totalImagesPullingTime="6.631449002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:42.193611779 +0000 UTC m=+74.024639342" watchObservedRunningTime="2026-09-10 23:37:42.19444194 +0000 UTC m=+74.025469472"
Sep 10 23:37:42.219574 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.219531    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2jz8n" podStartSLOduration=33.906281159 podStartE2EDuration="39.219518756s" totalImagesPullingTime="5.313237597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:42.219082444 +0000 UTC m=+74.050109996" watchObservedRunningTime="2026-09-10 23:37:42.219518756 +0000 UTC m=+74.050546290"
Sep 10 23:37:42.236964 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.236908    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9b7h5" podStartSLOduration=47.006871364 podStartE2EDuration="52.23689767s" totalImagesPullingTime="5.230026306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:42.235854525 +0000 UTC m=+74.066882053" watchObservedRunningTime="2026-09-10 23:37:42.23689767 +0000 UTC m=+74.067925204"
Sep 10 23:37:42.248472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.248447    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:42.483109 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:42.483076    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:45.077186 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:45.077153    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:45.177644 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:45.177616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:45.244405 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:45.244378    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:46.200131 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:46.200097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:46.273641 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:46.273597    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:46.656767 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:46.656699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:47.978179 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:47.978150    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.090485 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:48.090454    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.472117 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:48.472086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.577974 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:48.577944    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:48.838427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:48.838342    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" podUID="fb13d7b5-b5ba-40b6-af79-f20db76cf277" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 23:37:48.838427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:48.838405    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb"
Sep 10 23:37:48.838863 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:48.838833    2562 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a05b31423310c85316527addb0ba405531a55cad55240becff68a82350d9df7b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 10 23:37:48.838903 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:48.838884    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" podUID="fb13d7b5-b5ba-40b6-af79-f20db76cf277" containerName="service-proxy" containerID="cri-o://a05b31423310c85316527addb0ba405531a55cad55240becff68a82350d9df7b" gracePeriod=30
Sep 10 23:37:49.133761 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:49.133699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:49.187051 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:49.187026    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb13d7b5-b5ba-40b6-af79-f20db76cf277" containerID="a05b31423310c85316527addb0ba405531a55cad55240becff68a82350d9df7b" exitCode=2
Sep 10 23:37:49.187166 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:49.187100    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" event={"ID":"fb13d7b5-b5ba-40b6-af79-f20db76cf277","Type":"ContainerDied","Data":"a05b31423310c85316527addb0ba405531a55cad55240becff68a82350d9df7b"}
Sep 10 23:37:49.187166 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:49.187143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-765895748b-q2nrb" event={"ID":"fb13d7b5-b5ba-40b6-af79-f20db76cf277","Type":"ContainerStarted","Data":"53ba785e1bd8b3fe63561484dbe6a24e1023fc123c6405ff80eaef77c4adcbc7"}
Sep 10 23:37:49.261598 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:49.261576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:49.765395 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:49.765366    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:49.866470 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:49.866442    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.374830 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.374728    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.483102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.483071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.604113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.604083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.628401 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.628345    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rgrnb"]
Sep 10 23:37:51.630837 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.630821    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.633185 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.633168    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 23:37:51.633263 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.633209    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 23:37:51.633485 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.633466    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 23:37:51.634299 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.634277    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k96nt\""
Sep 10 23:37:51.634386 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.634279    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 23:37:51.636420 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.636519 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-tls\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.636570 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-wtmp\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.636665 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-root\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.636704 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-metrics-client-ca\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.636704 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636690    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-textfile\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.636781 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.636781 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-sys\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.636881 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.636792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jng9f\" (UniqueName: \"kubernetes.io/projected/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-kube-api-access-jng9f\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.696813 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.696787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.738006 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.737984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-textfile\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738097 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738097 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-sys\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738097 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jng9f\" (UniqueName: \"kubernetes.io/projected/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-kube-api-access-jng9f\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738097 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738119    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-tls\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-sys\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-wtmp\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-root\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-metrics-client-ca\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738254    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-wtmp\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738301 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:51.738296    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 23:37:51.738622 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738301    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-root\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738622 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-textfile\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738622 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:37:51.738367    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-tls podName:0e91ccb5-538d-4429-9afa-501c7d9b8f6d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:52.238335256 +0000 UTC m=+84.069362767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-tls") pod "node-exporter-rgrnb" (UID: "0e91ccb5-538d-4429-9afa-501c7d9b8f6d") : secret "node-exporter-tls" not found
Sep 10 23:37:51.738791 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738639    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.738791 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.738781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-metrics-client-ca\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.740453 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.740435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.749662 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.749642    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jng9f\" (UniqueName: \"kubernetes.io/projected/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-kube-api-access-jng9f\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:51.896978 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.896893    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:51.997075 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:51.997043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:52.165877 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:52.165804    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2jz8n"
Sep 10 23:37:52.242377 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:52.242346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-tls\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:52.244768 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:52.244739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e91ccb5-538d-4429-9afa-501c7d9b8f6d-node-exporter-tls\") pod \"node-exporter-rgrnb\" (UID: \"0e91ccb5-538d-4429-9afa-501c7d9b8f6d\") " pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:52.540184 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:52.540154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgrnb"
Sep 10 23:37:52.552144 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:37:52.552118    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e91ccb5_538d_4429_9afa_501c7d9b8f6d.slice/crio-360e8695802686c76b273f6ce3d7db89b89ee907ede589e293ac8f8a4c3da7ca WatchSource:0}: Error finding container 360e8695802686c76b273f6ce3d7db89b89ee907ede589e293ac8f8a4c3da7ca: Status 404 returned error can't find the container with id 360e8695802686c76b273f6ce3d7db89b89ee907ede589e293ac8f8a4c3da7ca
Sep 10 23:37:52.574799 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:52.574769    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:52.674490 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:52.674461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:52.739854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:52.739828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:53.153589 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:53.153553    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:53.204194 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:53.204163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgrnb" event={"ID":"0e91ccb5-538d-4429-9afa-501c7d9b8f6d","Type":"ContainerStarted","Data":"360e8695802686c76b273f6ce3d7db89b89ee907ede589e293ac8f8a4c3da7ca"}
Sep 10 23:37:54.088405 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:54.088374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:54.189017 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:54.188991    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:54.211936 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:54.211904    2562 generic.go:356] "Generic (PLEG): container finished" podID="0e91ccb5-538d-4429-9afa-501c7d9b8f6d" containerID="0f5b87fe1344fc5efede894848df35b96f2a81d1a8aae055ed9bcb94a7b943b0" exitCode=0
Sep 10 23:37:54.212069 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:54.211974    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgrnb" event={"ID":"0e91ccb5-538d-4429-9afa-501c7d9b8f6d","Type":"ContainerDied","Data":"0f5b87fe1344fc5efede894848df35b96f2a81d1a8aae055ed9bcb94a7b943b0"}
Sep 10 23:37:54.275570 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:54.275541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:54.333542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:54.333516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:54.605334 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:54.605257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:54.706120 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:54.706093    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:55.105609 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:55.105582    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:55.218695 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:55.218665    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgrnb" event={"ID":"0e91ccb5-538d-4429-9afa-501c7d9b8f6d","Type":"ContainerStarted","Data":"19c3b5af7f84d686e646bbe45cf42d1db993fda4cdfce01f616e13263faa79f0"}
Sep 10 23:37:55.218832 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:55.218722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgrnb" event={"ID":"0e91ccb5-538d-4429-9afa-501c7d9b8f6d","Type":"ContainerStarted","Data":"e0097dd5845fac5795c02ac57447e9c55358e52f6424b303f3aecd8c07e9b0cb"}
Sep 10 23:37:55.247600 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:55.247548    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rgrnb" podStartSLOduration=3.356865343 podStartE2EDuration="4.247533423s" totalImagesPullingTime="827.670405ms" totalInitContainerRuntime="62.997675ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:55.246361948 +0000 UTC m=+87.077389500" watchObservedRunningTime="2026-09-10 23:37:55.247533423 +0000 UTC m=+87.078561017"
Sep 10 23:37:55.283160 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:55.283139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:56.373235 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:56.373196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:56.480319 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:56.480285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:57.131981 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:57.131946    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:57.953476 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:57.953446    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"]
Sep 10 23:37:57.957698 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:57.957677    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:37:57.987277 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:57.987251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:58.056907 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:58.056885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:58.210963 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:58.210874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:58.311595 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:58.311559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:58.560579 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:58.560499    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:37:58.626899 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:37:58.626870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:01.108618 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:01.108574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:01.209457 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:01.209425    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:01.309672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:01.309643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:01.364839 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:01.364772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:03.753915 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:03.753878    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:03.805672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:03.805643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:06.108550 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:06.108522    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-27vsj"
Sep 10 23:38:07.746426 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:07.746394    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:07.847697 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:07.847670    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:07.947735 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:07.947707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:08.148591 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:08.148509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:08.249665 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:08.249642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:09.080898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:09.080864    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:09.137132 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:09.137096    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:22.972537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:22.972481    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94" podUID="576b260f-d773-4bd1-b98b-fdc0d99626d5" containerName="registry" containerID="cri-o://281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc" gracePeriod=30
Sep 10 23:38:23.205961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.205920    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:38:23.262150 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262093    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") pod \"576b260f-d773-4bd1-b98b-fdc0d99626d5\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") "
Sep 10 23:38:23.262150 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-certificates\") pod \"576b260f-d773-4bd1-b98b-fdc0d99626d5\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") "
Sep 10 23:38:23.262318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262165    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-installation-pull-secrets\") pod \"576b260f-d773-4bd1-b98b-fdc0d99626d5\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") "
Sep 10 23:38:23.262318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262193    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-trusted-ca\") pod \"576b260f-d773-4bd1-b98b-fdc0d99626d5\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") "
Sep 10 23:38:23.262318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262225    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh2ts\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-kube-api-access-qh2ts\") pod \"576b260f-d773-4bd1-b98b-fdc0d99626d5\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") "
Sep 10 23:38:23.262318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-bound-sa-token\") pod \"576b260f-d773-4bd1-b98b-fdc0d99626d5\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") "
Sep 10 23:38:23.262537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262325    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/576b260f-d773-4bd1-b98b-fdc0d99626d5-ca-trust-extracted\") pod \"576b260f-d773-4bd1-b98b-fdc0d99626d5\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") "
Sep 10 23:38:23.262537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262507    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-image-registry-private-configuration\") pod \"576b260f-d773-4bd1-b98b-fdc0d99626d5\" (UID: \"576b260f-d773-4bd1-b98b-fdc0d99626d5\") "
Sep 10 23:38:23.262881 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "576b260f-d773-4bd1-b98b-fdc0d99626d5" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:38:23.262881 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262714    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "576b260f-d773-4bd1-b98b-fdc0d99626d5" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:38:23.262881 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262826    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-trusted-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:38:23.262881 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.262844    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-certificates\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:38:23.264620 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.264598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "576b260f-d773-4bd1-b98b-fdc0d99626d5" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:38:23.264737 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.264626    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "576b260f-d773-4bd1-b98b-fdc0d99626d5" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:23.264973 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.264945    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-kube-api-access-qh2ts" (OuterVolumeSpecName: "kube-api-access-qh2ts") pod "576b260f-d773-4bd1-b98b-fdc0d99626d5" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5"). InnerVolumeSpecName "kube-api-access-qh2ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:23.265046 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.265014    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "576b260f-d773-4bd1-b98b-fdc0d99626d5" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:38:23.265147 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.265128    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "576b260f-d773-4bd1-b98b-fdc0d99626d5" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:23.271010 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.270989    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576b260f-d773-4bd1-b98b-fdc0d99626d5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "576b260f-d773-4bd1-b98b-fdc0d99626d5" (UID: "576b260f-d773-4bd1-b98b-fdc0d99626d5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:38:23.325036 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.325012    2562 generic.go:356] "Generic (PLEG): container finished" podID="576b260f-d773-4bd1-b98b-fdc0d99626d5" containerID="281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc" exitCode=0
Sep 10 23:38:23.325134 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.325058    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"
Sep 10 23:38:23.325134 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.325096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94" event={"ID":"576b260f-d773-4bd1-b98b-fdc0d99626d5","Type":"ContainerDied","Data":"281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc"}
Sep 10 23:38:23.325238 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.325152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6c8fc5d7-jkp94" event={"ID":"576b260f-d773-4bd1-b98b-fdc0d99626d5","Type":"ContainerDied","Data":"72d3ea5b5f2f1c8637b548f15a8f3de146f0c47b638285d559833eb29f499106"}
Sep 10 23:38:23.325238 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.325173    2562 scope.go:117] "RemoveContainer" containerID="281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc"
Sep 10 23:38:23.332803 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.332785    2562 scope.go:117] "RemoveContainer" containerID="281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc"
Sep 10 23:38:23.333106 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:38:23.333087    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc\": container with ID starting with 281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc not found: ID does not exist" containerID="281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc"
Sep 10 23:38:23.333173 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.333114    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc"} err="failed to get container status \"281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc\": rpc error: code = NotFound desc = could not find container \"281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc\": container with ID starting with 281deb96b0882f2e1ef9fe25edf017751eb099020dc7a3782e9e59e36bc82ddc not found: ID does not exist"
Sep 10 23:38:23.344741 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.344718    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"]
Sep 10 23:38:23.348192 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.348173    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c6c8fc5d7-jkp94"]
Sep 10 23:38:23.363142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.363121    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-bound-sa-token\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:38:23.363142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.363142    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/576b260f-d773-4bd1-b98b-fdc0d99626d5-ca-trust-extracted\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:38:23.363262 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.363151    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-image-registry-private-configuration\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:38:23.363262 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.363160    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-registry-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:38:23.363262 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.363169    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/576b260f-d773-4bd1-b98b-fdc0d99626d5-installation-pull-secrets\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:38:23.363262 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:23.363177    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh2ts\" (UniqueName: \"kubernetes.io/projected/576b260f-d773-4bd1-b98b-fdc0d99626d5-kube-api-access-qh2ts\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:38:24.654023 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:24.653992    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="576b260f-d773-4bd1-b98b-fdc0d99626d5" path="/var/lib/kubelet/pods/576b260f-d773-4bd1-b98b-fdc0d99626d5/volumes"
Sep 10 23:38:33.362559 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:33.362487    2562 generic.go:356] "Generic (PLEG): container finished" podID="17c84d6c-7d15-4b25-be27-78c331ceb2ae" containerID="fd07c9c8550851d4e5218d7c3d5f6f71ce06f9b34388dfc39feebac547290bf2" exitCode=0
Sep 10 23:38:33.362559 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:33.362548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wdgcw" event={"ID":"17c84d6c-7d15-4b25-be27-78c331ceb2ae","Type":"ContainerDied","Data":"fd07c9c8550851d4e5218d7c3d5f6f71ce06f9b34388dfc39feebac547290bf2"}
Sep 10 23:38:33.363033 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:33.362800    2562 scope.go:117] "RemoveContainer" containerID="fd07c9c8550851d4e5218d7c3d5f6f71ce06f9b34388dfc39feebac547290bf2"
Sep 10 23:38:33.929581 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:33.929535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:33.980001 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:33.979962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:34.368258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:34.368226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wdgcw" event={"ID":"17c84d6c-7d15-4b25-be27-78c331ceb2ae","Type":"ContainerStarted","Data":"665e51fdec06731558a2e93e71a00a9fb70a85d05f7eafe1a7eef7f09bc9861f"}
Sep 10 23:38:39.084696 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:39.084356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:39.135508 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:39.135465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:43.403516 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:43.403484    2562 generic.go:356] "Generic (PLEG): container finished" podID="356212ea-1e0e-452a-8e4c-a270b6c46ff7" containerID="348b329243a298be91b2537937ef0cef2a3524d3dd97675121f705f785029918" exitCode=0
Sep 10 23:38:43.403875 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:43.403527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s" event={"ID":"356212ea-1e0e-452a-8e4c-a270b6c46ff7","Type":"ContainerDied","Data":"348b329243a298be91b2537937ef0cef2a3524d3dd97675121f705f785029918"}
Sep 10 23:38:43.403875 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:43.403765    2562 scope.go:117] "RemoveContainer" containerID="348b329243a298be91b2537937ef0cef2a3524d3dd97675121f705f785029918"
Sep 10 23:38:44.409081 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:44.409041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hgt5s" event={"ID":"356212ea-1e0e-452a-8e4c-a270b6c46ff7","Type":"ContainerStarted","Data":"bda03fe907cb2ca73fbdbbfa11a6e87fe71aa4c5c66c48eb41249f8a800fb4f9"}
Sep 10 23:38:46.198963 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:46.198914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:38:48.431105 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:48.431070    2562 generic.go:356] "Generic (PLEG): container finished" podID="96674fff-8032-41fb-b75e-ca10f7d60860" containerID="ddb1d4251c8c96469de9f547c87cfc85f8493e356b9fb42ef26d3e7a89c7b002" exitCode=0
Sep 10 23:38:48.431471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:48.431140    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw" event={"ID":"96674fff-8032-41fb-b75e-ca10f7d60860","Type":"ContainerDied","Data":"ddb1d4251c8c96469de9f547c87cfc85f8493e356b9fb42ef26d3e7a89c7b002"}
Sep 10 23:38:48.431471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:48.431469    2562 scope.go:117] "RemoveContainer" containerID="ddb1d4251c8c96469de9f547c87cfc85f8493e356b9fb42ef26d3e7a89c7b002"
Sep 10 23:38:49.436393 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:38:49.436359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69fsw" event={"ID":"96674fff-8032-41fb-b75e-ca10f7d60860","Type":"ContainerStarted","Data":"3bc32f25e94d31e0de8483286be49bf3f71d9bd97080f5fddb950ebbd1ad0abb"}
Sep 10 23:39:02.025230 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:02.025197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:10.866149 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:10.866112    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:11.127307 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:11.127232    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:19.044246 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:19.044215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:26.086377 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:26.086294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:26.186834 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:26.186804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:30.097087 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:30.097045    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:30.189606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:30.189574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:30.427049 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:30.426969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:30.478901 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:30.478876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:31.027227 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:31.027194    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:31.079602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:31.079571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:31.828059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:31.828020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:31.929356 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:31.929330    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:37.980623 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:37.980588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:38.032540 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:38.032491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:41.106299 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:41.106258    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:41.208142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:41.208111    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:41.261972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:41.261948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:44.425494 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:44.425455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:39:44.527006 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:39:44.526973    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:04.471795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:04.471759    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:04.573556 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:04.573524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:27.426608 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:27.426573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:27.520269 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:27.520242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:49.076220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:49.076185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:49.852023 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:49.851995    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:52.723813 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:52.723743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:52.823226 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:52.823196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:54.692515 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:54.692480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:54.810343 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:54.810312    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:54.983037 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:54.982960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:55.072838 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:55.072812    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:57.344493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:57.344457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:57.561415 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:57.561387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:59.681844 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:59.681807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:40:59.780077 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:40:59.780045    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:04.969189 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:04.969159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:07.220093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:07.220040    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:07.327067 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:07.327016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:13.402752 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:13.402718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:13.502654 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:13.502625    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:15.094538 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:15.094507    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:23.109490 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:23.109458    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:23.161381 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:23.161351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:28.537265 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:28.537236    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:41:28.537640 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:28.537239    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:41:28.547777 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:28.547756    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 23:41:31.120849 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:31.120819    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:32.101766 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.101732    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"]
Sep 10 23:41:32.102024 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.102011    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="576b260f-d773-4bd1-b98b-fdc0d99626d5" containerName="registry"
Sep 10 23:41:32.102078 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.102025    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="576b260f-d773-4bd1-b98b-fdc0d99626d5" containerName="registry"
Sep 10 23:41:32.102112 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.102080    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="576b260f-d773-4bd1-b98b-fdc0d99626d5" containerName="registry"
Sep 10 23:41:32.104723 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.104706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.109113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.109064    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 23:41:32.109113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.109084    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b879w\""
Sep 10 23:41:32.109113 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.109087    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 23:41:32.109379 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.109366    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 23:41:32.109907 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.109885    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 23:41:32.123425 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.123400    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"]
Sep 10 23:41:32.228204 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.228176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:32.243230 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.243211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc0ac29-8968-4977-9be0-dbc149d88477-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.243338 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.243253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc0ac29-8968-4977-9be0-dbc149d88477-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.243338 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.243277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbc0ac29-8968-4977-9be0-dbc149d88477-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.243338 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.243296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbc0ac29-8968-4977-9be0-dbc149d88477-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.243449 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.243353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp66l\" (UniqueName: \"kubernetes.io/projected/fbc0ac29-8968-4977-9be0-dbc149d88477-kube-api-access-qp66l\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.305447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.305416    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:32.344549 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.344526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbc0ac29-8968-4977-9be0-dbc149d88477-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.344627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.344561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp66l\" (UniqueName: \"kubernetes.io/projected/fbc0ac29-8968-4977-9be0-dbc149d88477-kube-api-access-qp66l\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.344627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.344583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc0ac29-8968-4977-9be0-dbc149d88477-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.344627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.344617    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc0ac29-8968-4977-9be0-dbc149d88477-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.344783 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.344644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbc0ac29-8968-4977-9be0-dbc149d88477-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.345090 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.345063    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbc0ac29-8968-4977-9be0-dbc149d88477-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.346980 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.346958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc0ac29-8968-4977-9be0-dbc149d88477-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.347088 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.347048    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbc0ac29-8968-4977-9be0-dbc149d88477-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.347167 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.347144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc0ac29-8968-4977-9be0-dbc149d88477-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.355695 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.355646    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp66l\" (UniqueName: \"kubernetes.io/projected/fbc0ac29-8968-4977-9be0-dbc149d88477-kube-api-access-qp66l\") pod \"opendatahub-operator-controller-manager-55d55f8464-gnjz8\" (UID: \"fbc0ac29-8968-4977-9be0-dbc149d88477\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.404295 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.404268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:32.415122 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.415103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:32.504351 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.504320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:32.604300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.604255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:32.696780 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.696755    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:32.697425 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.697403    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"]
Sep 10 23:41:32.700197 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:41:32.700173    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc0ac29_8968_4977_9be0_dbc149d88477.slice/crio-65d09108b0a613d8c6abd889b53c297d80624b7b6f3f5cc99a999c81428d78c6 WatchSource:0}: Error finding container 65d09108b0a613d8c6abd889b53c297d80624b7b6f3f5cc99a999c81428d78c6: Status 404 returned error can't find the container with id 65d09108b0a613d8c6abd889b53c297d80624b7b6f3f5cc99a999c81428d78c6
Sep 10 23:41:32.701973 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:32.701955    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:41:33.021461 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:33.021396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8" event={"ID":"fbc0ac29-8968-4977-9be0-dbc149d88477","Type":"ContainerStarted","Data":"65d09108b0a613d8c6abd889b53c297d80624b7b6f3f5cc99a999c81428d78c6"}
Sep 10 23:41:33.396768 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:33.396685    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:33.500698 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:33.500664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:36.037912 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:36.037877    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8" event={"ID":"fbc0ac29-8968-4977-9be0-dbc149d88477","Type":"ContainerStarted","Data":"8c4ffa28ea769504cfc537af85bdc2050153d8fcd53aea6a0eb87a6c8f347895"}
Sep 10 23:41:37.043181 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:37.043147    2562 generic.go:356] "Generic (PLEG): container finished" podID="fbc0ac29-8968-4977-9be0-dbc149d88477" containerID="8c4ffa28ea769504cfc537af85bdc2050153d8fcd53aea6a0eb87a6c8f347895" exitCode=0
Sep 10 23:41:37.043521 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:37.043207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8" event={"ID":"fbc0ac29-8968-4977-9be0-dbc149d88477","Type":"ContainerDied","Data":"8c4ffa28ea769504cfc537af85bdc2050153d8fcd53aea6a0eb87a6c8f347895"}
Sep 10 23:41:38.048559 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:38.048526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8" event={"ID":"fbc0ac29-8968-4977-9be0-dbc149d88477","Type":"ContainerStarted","Data":"bf175edc0e8b696f3479e21b657a06968dea6bd4735a60c4ad87792d010fa288"}
Sep 10 23:41:38.048989 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:38.048663    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:38.070073 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:38.070030    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8" podStartSLOduration=1.85929096 podStartE2EDuration="6.070016224s" totalImagesPullingTime="3.183702128s" totalInitContainerRuntime="1.027023136s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:38.067860439 +0000 UTC m=+309.898887993" watchObservedRunningTime="2026-09-10 23:41:38.070016224 +0000 UTC m=+309.901043819"
Sep 10 23:41:49.054751 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:49.054725    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-gnjz8"
Sep 10 23:41:51.354333 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:51.354299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:51.562693 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:51.562664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:52.067893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:52.067860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:52.329337 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:52.329259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:54.730838 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:54.730810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:54.853844 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:54.853813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:55.984906 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:55.984875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:59.893334 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:59.893300    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:41:59.991909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:41:59.991860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:00.190954 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:00.190899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:00.311076 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:00.311040    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:00.410951 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:00.410896    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:02.693640 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.693593    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fpw9m"]
Sep 10 23:42:02.697045 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.697024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:02.699944 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.699904    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8cgfx\""
Sep 10 23:42:02.700142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.699910    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 23:42:02.701396 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.701374    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fpw9m"]
Sep 10 23:42:02.713040 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.713015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:02.827805 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.827776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:02.868140 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.868111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa6a65e9-9daf-4ed1-a14d-a2d9007b103d-cert\") pod \"odh-model-controller-7998c67dc4-fpw9m\" (UID: \"aa6a65e9-9daf-4ed1-a14d-a2d9007b103d\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:02.868251 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.868145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95r9d\" (UniqueName: \"kubernetes.io/projected/aa6a65e9-9daf-4ed1-a14d-a2d9007b103d-kube-api-access-95r9d\") pod \"odh-model-controller-7998c67dc4-fpw9m\" (UID: \"aa6a65e9-9daf-4ed1-a14d-a2d9007b103d\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:02.919909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.919877    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:02.969291 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.969220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa6a65e9-9daf-4ed1-a14d-a2d9007b103d-cert\") pod \"odh-model-controller-7998c67dc4-fpw9m\" (UID: \"aa6a65e9-9daf-4ed1-a14d-a2d9007b103d\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:02.969291 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.969261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95r9d\" (UniqueName: \"kubernetes.io/projected/aa6a65e9-9daf-4ed1-a14d-a2d9007b103d-kube-api-access-95r9d\") pod \"odh-model-controller-7998c67dc4-fpw9m\" (UID: \"aa6a65e9-9daf-4ed1-a14d-a2d9007b103d\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:02.971576 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.971548    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa6a65e9-9daf-4ed1-a14d-a2d9007b103d-cert\") pod \"odh-model-controller-7998c67dc4-fpw9m\" (UID: \"aa6a65e9-9daf-4ed1-a14d-a2d9007b103d\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:02.977862 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:02.977841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95r9d\" (UniqueName: \"kubernetes.io/projected/aa6a65e9-9daf-4ed1-a14d-a2d9007b103d-kube-api-access-95r9d\") pod \"odh-model-controller-7998c67dc4-fpw9m\" (UID: \"aa6a65e9-9daf-4ed1-a14d-a2d9007b103d\") " pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:03.008976 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.008951    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:03.281310 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.281280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fpw9m"]
Sep 10 23:42:03.283596 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:42:03.283567    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6a65e9_9daf_4ed1_a14d_a2d9007b103d.slice/crio-0d0dbeb07a32aa4337197eddfe0cd8db2fdea5cf410c5cbc6377827da544f3d9 WatchSource:0}: Error finding container 0d0dbeb07a32aa4337197eddfe0cd8db2fdea5cf410c5cbc6377827da544f3d9: Status 404 returned error can't find the container with id 0d0dbeb07a32aa4337197eddfe0cd8db2fdea5cf410c5cbc6377827da544f3d9
Sep 10 23:42:03.304698 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.304672    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-psxg2"]
Sep 10 23:42:03.309138 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.309117    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:03.311532 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.311514    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 23:42:03.317522 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.317502    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-psxg2"]
Sep 10 23:42:03.325063 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.325042    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:03.474344 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.474311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-tls-certs\") pod \"model-serving-api-54d89bd5fc-psxg2\" (UID: \"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec\") " pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:03.474522 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.474369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w898\" (UniqueName: \"kubernetes.io/projected/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-kube-api-access-6w898\") pod \"model-serving-api-54d89bd5fc-psxg2\" (UID: \"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec\") " pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:03.575828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.575750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-tls-certs\") pod \"model-serving-api-54d89bd5fc-psxg2\" (UID: \"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec\") " pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:03.575828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.575802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w898\" (UniqueName: \"kubernetes.io/projected/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-kube-api-access-6w898\") pod \"model-serving-api-54d89bd5fc-psxg2\" (UID: \"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec\") " pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:03.576030 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:42:03.575911    2562 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 10 23:42:03.576030 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:42:03.575995    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-tls-certs podName:0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec nodeName:}" failed. No retries permitted until 2026-09-10 23:42:04.075977246 +0000 UTC m=+335.907004782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-tls-certs") pod "model-serving-api-54d89bd5fc-psxg2" (UID: "0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec") : secret "model-serving-api-tls" not found
Sep 10 23:42:03.584537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.584515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w898\" (UniqueName: \"kubernetes.io/projected/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-kube-api-access-6w898\") pod \"model-serving-api-54d89bd5fc-psxg2\" (UID: \"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec\") " pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:03.874663 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:03.874583    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:04.080805 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:04.080756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-tls-certs\") pod \"model-serving-api-54d89bd5fc-psxg2\" (UID: \"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec\") " pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:04.083819 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:04.083769    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec-tls-certs\") pod \"model-serving-api-54d89bd5fc-psxg2\" (UID: \"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec\") " pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:04.153302 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:04.153215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m" event={"ID":"aa6a65e9-9daf-4ed1-a14d-a2d9007b103d","Type":"ContainerStarted","Data":"0d0dbeb07a32aa4337197eddfe0cd8db2fdea5cf410c5cbc6377827da544f3d9"}
Sep 10 23:42:04.220023 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:04.219698    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:04.510406 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:04.510317    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-psxg2"]
Sep 10 23:42:04.512961 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:42:04.512916    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6950ff_78d3_461f_a9fe_0f2e7ec3c1ec.slice/crio-0f878f577a298fa24b5ce90e73523c5603d32088ac858d933078515971a899a6 WatchSource:0}: Error finding container 0f878f577a298fa24b5ce90e73523c5603d32088ac858d933078515971a899a6: Status 404 returned error can't find the container with id 0f878f577a298fa24b5ce90e73523c5603d32088ac858d933078515971a899a6
Sep 10 23:42:05.159494 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:05.159456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-psxg2" event={"ID":"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec","Type":"ContainerStarted","Data":"0f878f577a298fa24b5ce90e73523c5603d32088ac858d933078515971a899a6"}
Sep 10 23:42:05.719563 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:05.719528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:05.797007 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:05.796973    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:06.167156 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:06.167088    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m" event={"ID":"aa6a65e9-9daf-4ed1-a14d-a2d9007b103d","Type":"ContainerStarted","Data":"7a83495185c540ffdd444b5d158ba56af9fde8bea7070adeb7976598c636d347"}
Sep 10 23:42:06.167576 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:06.167312    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:06.188951 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:06.188889    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m" podStartSLOduration=1.488006108 podStartE2EDuration="4.188869407s" totalImagesPullingTime="2.700863299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:06.188149481 +0000 UTC m=+338.019177037" watchObservedRunningTime="2026-09-10 23:42:06.188869407 +0000 UTC m=+338.019896941"
Sep 10 23:42:08.182005 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:08.181967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-psxg2" event={"ID":"0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec","Type":"ContainerStarted","Data":"55194ec7589cbba387d187279c31699f36bfc71c577089e5e7026a7cb398f534"}
Sep 10 23:42:08.182413 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:08.182073    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:08.198018 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:08.197971    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-psxg2" podStartSLOduration=1.8820680250000001 podStartE2EDuration="5.197955923s" totalImagesPullingTime="3.315887898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:08.196588467 +0000 UTC m=+340.027616023" watchObservedRunningTime="2026-09-10 23:42:08.197955923 +0000 UTC m=+340.028983455"
Sep 10 23:42:09.716596 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.716562    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"]
Sep 10 23:42:09.719960 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.719922    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.722643 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.722606    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:42:09.723635 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.723617    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 23:42:09.723838 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.723821    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 23:42:09.723949 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.723882    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 23:42:09.723949 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.723900    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 23:42:09.724063 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.723973    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m2crj\""
Sep 10 23:42:09.728276 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.728256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6w8\" (UniqueName: \"kubernetes.io/projected/30fee751-9e1e-483a-9e5d-d7eed8c53a49-kube-api-access-vv6w8\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.728382 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.728290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30fee751-9e1e-483a-9e5d-d7eed8c53a49-cert\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.728382 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.728370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30fee751-9e1e-483a-9e5d-d7eed8c53a49-manager-config\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.728519 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.728404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30fee751-9e1e-483a-9e5d-d7eed8c53a49-metrics-cert\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.733169 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.733147    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"]
Sep 10 23:42:09.828807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.828772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30fee751-9e1e-483a-9e5d-d7eed8c53a49-manager-config\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.829001 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.828815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30fee751-9e1e-483a-9e5d-d7eed8c53a49-metrics-cert\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.829001 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.828851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6w8\" (UniqueName: \"kubernetes.io/projected/30fee751-9e1e-483a-9e5d-d7eed8c53a49-kube-api-access-vv6w8\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.829001 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.828872    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30fee751-9e1e-483a-9e5d-d7eed8c53a49-cert\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.829458 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.829433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30fee751-9e1e-483a-9e5d-d7eed8c53a49-manager-config\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.831361 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.831335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30fee751-9e1e-483a-9e5d-d7eed8c53a49-cert\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.831440 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.831380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30fee751-9e1e-483a-9e5d-d7eed8c53a49-metrics-cert\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:09.840445 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:09.840427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6w8\" (UniqueName: \"kubernetes.io/projected/30fee751-9e1e-483a-9e5d-d7eed8c53a49-kube-api-access-vv6w8\") pod \"lws-controller-manager-66bf4bb994-nkvt7\" (UID: \"30fee751-9e1e-483a-9e5d-d7eed8c53a49\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:10.029186 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:10.029110    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:10.302881 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:10.302855    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"]
Sep 10 23:42:10.305368 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:42:10.305343    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fee751_9e1e_483a_9e5d_d7eed8c53a49.slice/crio-14144d7440070547b1695fb1d2d5190878a5e7cee363fe28bd985ff8bac63bef WatchSource:0}: Error finding container 14144d7440070547b1695fb1d2d5190878a5e7cee363fe28bd985ff8bac63bef: Status 404 returned error can't find the container with id 14144d7440070547b1695fb1d2d5190878a5e7cee363fe28bd985ff8bac63bef
Sep 10 23:42:11.196428 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:11.196393    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7" event={"ID":"30fee751-9e1e-483a-9e5d-d7eed8c53a49","Type":"ContainerStarted","Data":"14144d7440070547b1695fb1d2d5190878a5e7cee363fe28bd985ff8bac63bef"}
Sep 10 23:42:12.752643 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:12.752612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:12.863093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:12.863049    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:13.152139 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:13.152044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:13.206936 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:13.206891    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7" event={"ID":"30fee751-9e1e-483a-9e5d-d7eed8c53a49","Type":"ContainerStarted","Data":"cf53ad67f1ef410347b0ef78c2a62759bd955285ccee13b60b65d78db06861b7"}
Sep 10 23:42:13.207086 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:13.207007    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:13.567148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:13.567117    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:14.029533 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:14.029501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:14.184091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:14.184058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:17.177038 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:17.177010    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fpw9m"
Sep 10 23:42:17.196501 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:17.196451    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7" podStartSLOduration=6.145313397 podStartE2EDuration="8.196432894s" totalImagesPullingTime="2.051119497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:13.231214674 +0000 UTC m=+345.062242225" watchObservedRunningTime="2026-09-10 23:42:17.196432894 +0000 UTC m=+349.027460428"
Sep 10 23:42:19.190355 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:19.190330    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-psxg2"
Sep 10 23:42:24.213844 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:24.213761    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-nkvt7"
Sep 10 23:42:27.441622 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:27.441579    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:43.686611 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:43.686571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:43.784457 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:43.784425    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:43.885253 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:43.885223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:44.415170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:44.415129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:44.471218 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:44.471194    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:55.093852 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:55.093805    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:55.197346 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:55.197317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:42:55.260579 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:42:55.260541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:02.188237 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:02.188196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:02.289315 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:02.289273    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:03.442492 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:03.442455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:03.555333 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:03.555298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:15.163655 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:15.163618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:17.308220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.308184    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"]
Sep 10 23:43:17.310201 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.310185    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:17.313409 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.313389    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 23:43:17.313657 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.313639    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 23:43:17.314467 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.314452    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9rvgk\""
Sep 10 23:43:17.334568 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.334546    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"]
Sep 10 23:43:17.418186 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.418166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlskl\" (UniqueName: \"kubernetes.io/projected/e03127f7-bb5c-4665-b3b6-1ed6c98145d4-kube-api-access-wlskl\") pod \"limitador-operator-controller-manager-74495c69f8-6j6cw\" (UID: \"e03127f7-bb5c-4665-b3b6-1ed6c98145d4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:17.519094 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.519068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlskl\" (UniqueName: \"kubernetes.io/projected/e03127f7-bb5c-4665-b3b6-1ed6c98145d4-kube-api-access-wlskl\") pod \"limitador-operator-controller-manager-74495c69f8-6j6cw\" (UID: \"e03127f7-bb5c-4665-b3b6-1ed6c98145d4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:17.533359 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.533337    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlskl\" (UniqueName: \"kubernetes.io/projected/e03127f7-bb5c-4665-b3b6-1ed6c98145d4-kube-api-access-wlskl\") pod \"limitador-operator-controller-manager-74495c69f8-6j6cw\" (UID: \"e03127f7-bb5c-4665-b3b6-1ed6c98145d4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:17.619887 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.619829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:17.927692 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:17.927624    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"]
Sep 10 23:43:17.931112 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:43:17.931086    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03127f7_bb5c_4665_b3b6_1ed6c98145d4.slice/crio-846f9b41c92c456d210b93245bcdc6a593023cc8ce0d04ea1e3b15f5cc61e6c2 WatchSource:0}: Error finding container 846f9b41c92c456d210b93245bcdc6a593023cc8ce0d04ea1e3b15f5cc61e6c2: Status 404 returned error can't find the container with id 846f9b41c92c456d210b93245bcdc6a593023cc8ce0d04ea1e3b15f5cc61e6c2
Sep 10 23:43:18.047028 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:18.047000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:18.470851 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:18.470820    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw" event={"ID":"e03127f7-bb5c-4665-b3b6-1ed6c98145d4","Type":"ContainerStarted","Data":"846f9b41c92c456d210b93245bcdc6a593023cc8ce0d04ea1e3b15f5cc61e6c2"}
Sep 10 23:43:20.169398 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:20.169337    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:20.302350 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:20.302304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:20.480586 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:20.480558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw" event={"ID":"e03127f7-bb5c-4665-b3b6-1ed6c98145d4","Type":"ContainerStarted","Data":"aa5e6c1834304629ab9667c5a36a11c06adfb5e98e613270ba4910a19bc4c374"}
Sep 10 23:43:20.480721 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:20.480704    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:20.536079 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:20.536037    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw" podStartSLOduration=1.155903712 podStartE2EDuration="3.536023524s" totalImagesPullingTime="2.380119812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:20.535670732 +0000 UTC m=+412.366698285" watchObservedRunningTime="2026-09-10 23:43:20.536023524 +0000 UTC m=+412.367051057"
Sep 10 23:43:20.823331 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:20.823251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:21.260143 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:21.260110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:21.786980 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:21.786949    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:22.088987 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:22.088901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:24.596563 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:24.596530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:25.048833 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:25.048799    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:25.805604 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:25.805571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:28.577991 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:28.577955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:28.678098 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:28.678069    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:30.953622 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:30.953588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:31.054338 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:31.054305    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:31.486348 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:31.486317    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:33.334215 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:33.334178    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:33.477431 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:33.477402    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:36.391644 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:36.391608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:36.490522 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:36.490490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:44.144134 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.144101    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"]
Sep 10 23:43:44.147342 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.147324    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.150188 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.150165    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5wwxc\""
Sep 10 23:43:44.176453 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.176433    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"]
Sep 10 23:43:44.269495 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.269465    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"]
Sep 10 23:43:44.269678 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:43:44.269659    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-7ppp2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72" podUID="88c109ff-f3e2-44c6-8974-0ff27e30b173"
Sep 10 23:43:44.315362 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.315338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c109ff-f3e2-44c6-8974-0ff27e30b173-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" (UID: \"88c109ff-f3e2-44c6-8974-0ff27e30b173\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.315467 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.315401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ppp2\" (UniqueName: \"kubernetes.io/projected/88c109ff-f3e2-44c6-8974-0ff27e30b173-kube-api-access-7ppp2\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" (UID: \"88c109ff-f3e2-44c6-8974-0ff27e30b173\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.416625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.416560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c109ff-f3e2-44c6-8974-0ff27e30b173-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" (UID: \"88c109ff-f3e2-44c6-8974-0ff27e30b173\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.416625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.416595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ppp2\" (UniqueName: \"kubernetes.io/projected/88c109ff-f3e2-44c6-8974-0ff27e30b173-kube-api-access-7ppp2\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" (UID: \"88c109ff-f3e2-44c6-8974-0ff27e30b173\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.416914 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.416885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c109ff-f3e2-44c6-8974-0ff27e30b173-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" (UID: \"88c109ff-f3e2-44c6-8974-0ff27e30b173\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.427756 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.427733    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"]
Sep 10 23:43:44.434466 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.434406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ppp2\" (UniqueName: \"kubernetes.io/projected/88c109ff-f3e2-44c6-8974-0ff27e30b173-kube-api-access-7ppp2\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" (UID: \"88c109ff-f3e2-44c6-8974-0ff27e30b173\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.435484 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.435461    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"]
Sep 10 23:43:44.455670 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.455644    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"]
Sep 10 23:43:44.455837 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.455816    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw" podUID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" containerName="manager" containerID="cri-o://aa5e6c1834304629ab9667c5a36a11c06adfb5e98e613270ba4910a19bc4c374" gracePeriod=2
Sep 10 23:43:44.477480 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.477457    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"]
Sep 10 23:43:44.479031 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:43:44.478986    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e03127f7-bb5c-4665-b3b6-1ed6c98145d4\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-6j6cw\": pods \"limitador-operator-controller-manager-74495c69f8-6j6cw\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:44.504124 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.504098    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"]
Sep 10 23:43:44.504536 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.504522    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" containerName="manager"
Sep 10 23:43:44.504578 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.504540    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" containerName="manager"
Sep 10 23:43:44.504641 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.504630    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" containerName="manager"
Sep 10 23:43:44.507940 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.507908    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"
Sep 10 23:43:44.510651 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:43:44.510620    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j6cw\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:44.517026 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.516920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tj5t\" (UniqueName: \"kubernetes.io/projected/947b5b3a-6a42-41b9-8680-c504b4299c7d-kube-api-access-4tj5t\") pod \"limitador-operator-controller-manager-74495c69f8-wdrx7\" (UID: \"947b5b3a-6a42-41b9-8680-c504b4299c7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"
Sep 10 23:43:44.528102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.528081    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"]
Sep 10 23:43:44.585417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.585370    2562 generic.go:356] "Generic (PLEG): container finished" podID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" containerID="aa5e6c1834304629ab9667c5a36a11c06adfb5e98e613270ba4910a19bc4c374" exitCode=0
Sep 10 23:43:44.585551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.585538    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.588583 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:43:44.588553    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="88c109ff-f3e2-44c6-8974-0ff27e30b173" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.590534 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:43:44.590510    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j6cw\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:44.597959 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.597913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.600206 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:43:44.600176    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="88c109ff-f3e2-44c6-8974-0ff27e30b173" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:44.602099 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:43:44.602064    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j6cw\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:44.617489 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.617460    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tj5t\" (UniqueName: \"kubernetes.io/projected/947b5b3a-6a42-41b9-8680-c504b4299c7d-kube-api-access-4tj5t\") pod \"limitador-operator-controller-manager-74495c69f8-wdrx7\" (UID: \"947b5b3a-6a42-41b9-8680-c504b4299c7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"
Sep 10 23:43:44.628463 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.628441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tj5t\" (UniqueName: \"kubernetes.io/projected/947b5b3a-6a42-41b9-8680-c504b4299c7d-kube-api-access-4tj5t\") pod \"limitador-operator-controller-manager-74495c69f8-wdrx7\" (UID: \"947b5b3a-6a42-41b9-8680-c504b4299c7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"
Sep 10 23:43:44.690091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.690068    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:44.718051 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.718030    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ppp2\" (UniqueName: \"kubernetes.io/projected/88c109ff-f3e2-44c6-8974-0ff27e30b173-kube-api-access-7ppp2\") pod \"88c109ff-f3e2-44c6-8974-0ff27e30b173\" (UID: \"88c109ff-f3e2-44c6-8974-0ff27e30b173\") "
Sep 10 23:43:44.718121 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.718072    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c109ff-f3e2-44c6-8974-0ff27e30b173-extensions-socket-volume\") pod \"88c109ff-f3e2-44c6-8974-0ff27e30b173\" (UID: \"88c109ff-f3e2-44c6-8974-0ff27e30b173\") "
Sep 10 23:43:44.718423 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.718389    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c109ff-f3e2-44c6-8974-0ff27e30b173-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "88c109ff-f3e2-44c6-8974-0ff27e30b173" (UID: "88c109ff-f3e2-44c6-8974-0ff27e30b173"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:43:44.720081 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.720062    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c109ff-f3e2-44c6-8974-0ff27e30b173-kube-api-access-7ppp2" (OuterVolumeSpecName: "kube-api-access-7ppp2") pod "88c109ff-f3e2-44c6-8974-0ff27e30b173" (UID: "88c109ff-f3e2-44c6-8974-0ff27e30b173"). InnerVolumeSpecName "kube-api-access-7ppp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:43:44.818729 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.818704    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlskl\" (UniqueName: \"kubernetes.io/projected/e03127f7-bb5c-4665-b3b6-1ed6c98145d4-kube-api-access-wlskl\") pod \"e03127f7-bb5c-4665-b3b6-1ed6c98145d4\" (UID: \"e03127f7-bb5c-4665-b3b6-1ed6c98145d4\") "
Sep 10 23:43:44.818854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.818843    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c109ff-f3e2-44c6-8974-0ff27e30b173-extensions-socket-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:43:44.818900 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.818857    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ppp2\" (UniqueName: \"kubernetes.io/projected/88c109ff-f3e2-44c6-8974-0ff27e30b173-kube-api-access-7ppp2\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:43:44.820623 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.820602    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03127f7-bb5c-4665-b3b6-1ed6c98145d4-kube-api-access-wlskl" (OuterVolumeSpecName: "kube-api-access-wlskl") pod "e03127f7-bb5c-4665-b3b6-1ed6c98145d4" (UID: "e03127f7-bb5c-4665-b3b6-1ed6c98145d4"). InnerVolumeSpecName "kube-api-access-wlskl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:43:44.822772 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.822756    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"
Sep 10 23:43:44.919426 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:44.919399    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlskl\" (UniqueName: \"kubernetes.io/projected/e03127f7-bb5c-4665-b3b6-1ed6c98145d4-kube-api-access-wlskl\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:43:45.158436 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.158283    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"]
Sep 10 23:43:45.161145 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:43:45.161122    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947b5b3a_6a42_41b9_8680_c504b4299c7d.slice/crio-974f8713265cc14b543f6398fd7a1006d6cdf61ca6f32c680ba32be9723d9471 WatchSource:0}: Error finding container 974f8713265cc14b543f6398fd7a1006d6cdf61ca6f32c680ba32be9723d9471: Status 404 returned error can't find the container with id 974f8713265cc14b543f6398fd7a1006d6cdf61ca6f32c680ba32be9723d9471
Sep 10 23:43:45.374161 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.374125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:45.486793 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.486766    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:45.591505 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.591483    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j6cw"
Sep 10 23:43:45.591505 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.591500    2562 scope.go:117] "RemoveContainer" containerID="aa5e6c1834304629ab9667c5a36a11c06adfb5e98e613270ba4910a19bc4c374"
Sep 10 23:43:45.593610 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.593569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7" event={"ID":"947b5b3a-6a42-41b9-8680-c504b4299c7d","Type":"ContainerStarted","Data":"9574b7aa4b3f82a24903da746bdce6324bb31af22e1c41743d4062a1b5ea8c0d"}
Sep 10 23:43:45.593727 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.593608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:45.593788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.593739    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"
Sep 10 23:43:45.593788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.593773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7" event={"ID":"947b5b3a-6a42-41b9-8680-c504b4299c7d","Type":"ContainerStarted","Data":"974f8713265cc14b543f6398fd7a1006d6cdf61ca6f32c680ba32be9723d9471"}
Sep 10 23:43:45.595911 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.595888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:45.621746 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:43:45.621711    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-74v72\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="88c109ff-f3e2-44c6-8974-0ff27e30b173" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-74v72"
Sep 10 23:43:45.622352 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:45.622317    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7" podStartSLOduration=1.6223046509999999 podStartE2EDuration="1.622304651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:45.619737973 +0000 UTC m=+437.450765505" watchObservedRunningTime="2026-09-10 23:43:45.622304651 +0000 UTC m=+437.453332183"
Sep 10 23:43:46.654343 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:46.654310    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c109ff-f3e2-44c6-8974-0ff27e30b173" path="/var/lib/kubelet/pods/88c109ff-f3e2-44c6-8974-0ff27e30b173/volumes"
Sep 10 23:43:46.654694 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:46.654516    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e03127f7-bb5c-4665-b3b6-1ed6c98145d4" path="/var/lib/kubelet/pods/e03127f7-bb5c-4665-b3b6-1ed6c98145d4/volumes"
Sep 10 23:43:47.838174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:47.838143    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:47.944278 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:47.944235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:50.589364 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:50.589334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:50.706209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:50.706182    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:55.529325 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:55.529296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:55.629214 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:55.629184    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:55.729410 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:55.729376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:55.834086 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:55.834014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:55.905819 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:55.905793    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:43:56.601249 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:43:56.601221    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wdrx7"
Sep 10 23:44:04.211099 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:04.211054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:04.310417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:04.310386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:10.281691 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:10.281658    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:16.754493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:16.754461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:17.139226 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:17.139124    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:19.700240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:19.700205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:19.758010 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:19.757985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:21.527097 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.527065    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wtzfk"]
Sep 10 23:44:21.529528 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.529501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:21.533848 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.533830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:21.537034 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.537012    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 23:44:21.537197 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.537183    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fdzdl\""
Sep 10 23:44:21.541682 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.541666    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wtzfk"]
Sep 10 23:44:21.579657 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.579636    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h77t\" (UniqueName: \"kubernetes.io/projected/a4e2c65b-5080-4661-9980-d69bbf5e27d8-kube-api-access-6h77t\") pod \"limitador-limitador-8588749977-wtzfk\" (UID: \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\") " pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:21.579750 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.579680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4e2c65b-5080-4661-9980-d69bbf5e27d8-config-file\") pod \"limitador-limitador-8588749977-wtzfk\" (UID: \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\") " pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:21.618843 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.618812    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wtzfk"]
Sep 10 23:44:21.629528 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.629505    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:21.680787 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.680760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4e2c65b-5080-4661-9980-d69bbf5e27d8-config-file\") pod \"limitador-limitador-8588749977-wtzfk\" (UID: \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\") " pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:21.680877 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.680819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h77t\" (UniqueName: \"kubernetes.io/projected/a4e2c65b-5080-4661-9980-d69bbf5e27d8-kube-api-access-6h77t\") pod \"limitador-limitador-8588749977-wtzfk\" (UID: \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\") " pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:21.681417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.681398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4e2c65b-5080-4661-9980-d69bbf5e27d8-config-file\") pod \"limitador-limitador-8588749977-wtzfk\" (UID: \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\") " pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:21.688895 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.688875    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h77t\" (UniqueName: \"kubernetes.io/projected/a4e2c65b-5080-4661-9980-d69bbf5e27d8-kube-api-access-6h77t\") pod \"limitador-limitador-8588749977-wtzfk\" (UID: \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\") " pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:21.734354 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.734326    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:21.845032 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:21.844979    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:22.113602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.113574    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wtzfk"]
Sep 10 23:44:22.116456 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:44:22.116433    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e2c65b_5080_4661_9980_d69bbf5e27d8.slice/crio-b60cfddddab2a2e67bb3ee8689b5ddc48cda5cb66367705e7021fb1bfd0a6365 WatchSource:0}: Error finding container b60cfddddab2a2e67bb3ee8689b5ddc48cda5cb66367705e7021fb1bfd0a6365: Status 404 returned error can't find the container with id b60cfddddab2a2e67bb3ee8689b5ddc48cda5cb66367705e7021fb1bfd0a6365
Sep 10 23:44:22.339718 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.339682    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kqcdm"]
Sep 10 23:44:22.342794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.342774    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm"
Sep 10 23:44:22.345529 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.345512    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rcb9d\""
Sep 10 23:44:22.353949 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.353906    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kqcdm"]
Sep 10 23:44:22.386942 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.386882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp55t\" (UniqueName: \"kubernetes.io/projected/f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83-kube-api-access-hp55t\") pod \"authorino-7bcfd8f9cd-kqcdm\" (UID: \"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm"
Sep 10 23:44:22.393702 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.393682    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:22.487872 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.487844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp55t\" (UniqueName: \"kubernetes.io/projected/f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83-kube-api-access-hp55t\") pod \"authorino-7bcfd8f9cd-kqcdm\" (UID: \"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm"
Sep 10 23:44:22.493882 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.493853    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:22.500426 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.500400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp55t\" (UniqueName: \"kubernetes.io/projected/f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83-kube-api-access-hp55t\") pod \"authorino-7bcfd8f9cd-kqcdm\" (UID: \"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83\") " pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm"
Sep 10 23:44:22.651766 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.651698    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm"
Sep 10 23:44:22.753809 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.753775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk" event={"ID":"a4e2c65b-5080-4661-9980-d69bbf5e27d8","Type":"ContainerStarted","Data":"b60cfddddab2a2e67bb3ee8689b5ddc48cda5cb66367705e7021fb1bfd0a6365"}
Sep 10 23:44:22.919057 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:22.918998    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kqcdm"]
Sep 10 23:44:23.084556 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:44:23.084526    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f29e8e_5424_4e4f_ba49_ec7d77ce5b83.slice/crio-c670f53a89d1b2a9f225f41add88fa85fdf8e957316ab8113896da07446c748b WatchSource:0}: Error finding container c670f53a89d1b2a9f225f41add88fa85fdf8e957316ab8113896da07446c748b: Status 404 returned error can't find the container with id c670f53a89d1b2a9f225f41add88fa85fdf8e957316ab8113896da07446c748b
Sep 10 23:44:23.759965 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:23.759915    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm" event={"ID":"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83","Type":"ContainerStarted","Data":"c670f53a89d1b2a9f225f41add88fa85fdf8e957316ab8113896da07446c748b"}
Sep 10 23:44:23.761310 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:23.761288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk" event={"ID":"a4e2c65b-5080-4661-9980-d69bbf5e27d8","Type":"ContainerStarted","Data":"f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd"}
Sep 10 23:44:23.761517 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:23.761499    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:23.779184 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:23.779139    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk" podStartSLOduration=1.754023878 podStartE2EDuration="2.779125755s" totalImagesPullingTime="1.025101877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:44:23.778136673 +0000 UTC m=+475.609164226" watchObservedRunningTime="2026-09-10 23:44:23.779125755 +0000 UTC m=+475.610153287"
Sep 10 23:44:25.553116 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:25.553076    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kqcdm"]
Sep 10 23:44:25.771598 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:25.771556    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm" event={"ID":"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83","Type":"ContainerStarted","Data":"a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f"}
Sep 10 23:44:26.775222 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:26.775174    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm" podUID="f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83" containerName="authorino" containerID="cri-o://a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f" gracePeriod=30
Sep 10 23:44:26.794447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:26.794405    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm" podStartSLOduration=3.065471053 podStartE2EDuration="4.794388195s" totalImagesPullingTime="1.728917142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:44:25.786494951 +0000 UTC m=+477.617522493" watchObservedRunningTime="2026-09-10 23:44:26.794388195 +0000 UTC m=+478.625415710"
Sep 10 23:44:27.014487 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.014465    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm"
Sep 10 23:44:27.124528 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.124474    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp55t\" (UniqueName: \"kubernetes.io/projected/f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83-kube-api-access-hp55t\") pod \"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83\" (UID: \"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83\") "
Sep 10 23:44:27.126383 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.126349    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83-kube-api-access-hp55t" (OuterVolumeSpecName: "kube-api-access-hp55t") pod "f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83" (UID: "f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83"). InnerVolumeSpecName "kube-api-access-hp55t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:44:27.225230 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.225205    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp55t\" (UniqueName: \"kubernetes.io/projected/f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83-kube-api-access-hp55t\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:44:27.780280 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.780246    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83" containerID="a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f" exitCode=0
Sep 10 23:44:27.780653 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.780285    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm"
Sep 10 23:44:27.780653 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.780324    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm" event={"ID":"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83","Type":"ContainerDied","Data":"a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f"}
Sep 10 23:44:27.780653 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.780361    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-kqcdm" event={"ID":"f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83","Type":"ContainerDied","Data":"c670f53a89d1b2a9f225f41add88fa85fdf8e957316ab8113896da07446c748b"}
Sep 10 23:44:27.780653 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.780377    2562 scope.go:117] "RemoveContainer" containerID="a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f"
Sep 10 23:44:27.789077 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.789057    2562 scope.go:117] "RemoveContainer" containerID="a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f"
Sep 10 23:44:27.789334 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:44:27.789312    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f\": container with ID starting with a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f not found: ID does not exist" containerID="a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f"
Sep 10 23:44:27.789391 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.789344    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f"} err="failed to get container status \"a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f\": rpc error: code = NotFound desc = could not find container \"a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f\": container with ID starting with a9d0730cae37ee3bab0dbf2c96b3311721c499413e599b9e4036597161c0fa1f not found: ID does not exist"
Sep 10 23:44:27.801100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.801078    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kqcdm"]
Sep 10 23:44:27.805006 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:27.804986    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-kqcdm"]
Sep 10 23:44:28.654791 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:28.654759    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83" path="/var/lib/kubelet/pods/f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83/volumes"
Sep 10 23:44:30.252638 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:30.252606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:30.313156 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:30.313128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:34.766218 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:34.766185    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:35.563227 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:35.563197    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wtzfk"]
Sep 10 23:44:35.563497 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:35.563364    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk" podUID="a4e2c65b-5080-4661-9980-d69bbf5e27d8" containerName="limitador" containerID="cri-o://f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd" gracePeriod=30
Sep 10 23:44:36.105853 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.105831    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:36.189313 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.189290    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h77t\" (UniqueName: \"kubernetes.io/projected/a4e2c65b-5080-4661-9980-d69bbf5e27d8-kube-api-access-6h77t\") pod \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\" (UID: \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\") "
Sep 10 23:44:36.189433 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.189362    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4e2c65b-5080-4661-9980-d69bbf5e27d8-config-file\") pod \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\" (UID: \"a4e2c65b-5080-4661-9980-d69bbf5e27d8\") "
Sep 10 23:44:36.189694 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.189663    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e2c65b-5080-4661-9980-d69bbf5e27d8-config-file" (OuterVolumeSpecName: "config-file") pod "a4e2c65b-5080-4661-9980-d69bbf5e27d8" (UID: "a4e2c65b-5080-4661-9980-d69bbf5e27d8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:44:36.191307 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.191279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e2c65b-5080-4661-9980-d69bbf5e27d8-kube-api-access-6h77t" (OuterVolumeSpecName: "kube-api-access-6h77t") pod "a4e2c65b-5080-4661-9980-d69bbf5e27d8" (UID: "a4e2c65b-5080-4661-9980-d69bbf5e27d8"). InnerVolumeSpecName "kube-api-access-6h77t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:44:36.290375 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.290350    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h77t\" (UniqueName: \"kubernetes.io/projected/a4e2c65b-5080-4661-9980-d69bbf5e27d8-kube-api-access-6h77t\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:44:36.290375 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.290372    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4e2c65b-5080-4661-9980-d69bbf5e27d8-config-file\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:44:36.824913 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.824883    2562 generic.go:356] "Generic (PLEG): container finished" podID="a4e2c65b-5080-4661-9980-d69bbf5e27d8" containerID="f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd" exitCode=0
Sep 10 23:44:36.825091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.824953    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk"
Sep 10 23:44:36.825091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.824973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk" event={"ID":"a4e2c65b-5080-4661-9980-d69bbf5e27d8","Type":"ContainerDied","Data":"f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd"}
Sep 10 23:44:36.825091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.825050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wtzfk" event={"ID":"a4e2c65b-5080-4661-9980-d69bbf5e27d8","Type":"ContainerDied","Data":"b60cfddddab2a2e67bb3ee8689b5ddc48cda5cb66367705e7021fb1bfd0a6365"}
Sep 10 23:44:36.825091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.825073    2562 scope.go:117] "RemoveContainer" containerID="f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd"
Sep 10 23:44:36.833442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.833423    2562 scope.go:117] "RemoveContainer" containerID="f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd"
Sep 10 23:44:36.833712 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:44:36.833681    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd\": container with ID starting with f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd not found: ID does not exist" containerID="f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd"
Sep 10 23:44:36.833778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.833722    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd"} err="failed to get container status \"f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd\": rpc error: code = NotFound desc = could not find container \"f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd\": container with ID starting with f5e25c8a635a51280bf72b777363fbf832cafea9b2a4421a8beeb94fbd5c7fbd not found: ID does not exist"
Sep 10 23:44:36.842160 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.842139    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wtzfk"]
Sep 10 23:44:36.857065 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:36.857043    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wtzfk"]
Sep 10 23:44:37.634918 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:37.634886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:37.731442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:37.731411    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:38.102661 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.102629    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"]
Sep 10 23:44:38.102959 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.102945    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e2c65b-5080-4661-9980-d69bbf5e27d8" containerName="limitador"
Sep 10 23:44:38.102959 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.102959    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e2c65b-5080-4661-9980-d69bbf5e27d8" containerName="limitador"
Sep 10 23:44:38.103060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.102974    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83" containerName="authorino"
Sep 10 23:44:38.103060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.102980    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83" containerName="authorino"
Sep 10 23:44:38.103060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.103025    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f29e8e-5424-4e4f-ba49-ec7d77ce5b83" containerName="authorino"
Sep 10 23:44:38.103060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.103043    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e2c65b-5080-4661-9980-d69bbf5e27d8" containerName="limitador"
Sep 10 23:44:38.107320 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.107287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:38.110584 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.110561    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 23:44:38.111065 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.111041    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-k2msc\""
Sep 10 23:44:38.111551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.111529    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:44:38.111667 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.111649    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:44:38.118739 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.118714    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"]
Sep 10 23:44:38.153664 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.153624    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:38.205487 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.205454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a1599a9-3e16-4923-9482-2155fdeebff9-data\") pod \"postgres-69fb4594cf-sr9bv\" (UID: \"9a1599a9-3e16-4923-9482-2155fdeebff9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:38.205584 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.205528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwpt\" (UniqueName: \"kubernetes.io/projected/9a1599a9-3e16-4923-9482-2155fdeebff9-kube-api-access-vjwpt\") pod \"postgres-69fb4594cf-sr9bv\" (UID: \"9a1599a9-3e16-4923-9482-2155fdeebff9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:38.253235 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.253212    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:38.306750 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.306728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwpt\" (UniqueName: \"kubernetes.io/projected/9a1599a9-3e16-4923-9482-2155fdeebff9-kube-api-access-vjwpt\") pod \"postgres-69fb4594cf-sr9bv\" (UID: \"9a1599a9-3e16-4923-9482-2155fdeebff9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:38.306836 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.306781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a1599a9-3e16-4923-9482-2155fdeebff9-data\") pod \"postgres-69fb4594cf-sr9bv\" (UID: \"9a1599a9-3e16-4923-9482-2155fdeebff9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:38.307132 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.307116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a1599a9-3e16-4923-9482-2155fdeebff9-data\") pod \"postgres-69fb4594cf-sr9bv\" (UID: \"9a1599a9-3e16-4923-9482-2155fdeebff9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:38.327656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.327631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwpt\" (UniqueName: \"kubernetes.io/projected/9a1599a9-3e16-4923-9482-2155fdeebff9-kube-api-access-vjwpt\") pod \"postgres-69fb4594cf-sr9bv\" (UID: \"9a1599a9-3e16-4923-9482-2155fdeebff9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:38.420264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.420217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:38.654321 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.654290    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4e2c65b-5080-4661-9980-d69bbf5e27d8" path="/var/lib/kubelet/pods/a4e2c65b-5080-4661-9980-d69bbf5e27d8/volumes"
Sep 10 23:44:38.687228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.687206    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"]
Sep 10 23:44:38.689509 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:44:38.689480    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1599a9_3e16_4923_9482_2155fdeebff9.slice/crio-c2ed5a264cfe779495e2ff575dc906bc0e54b86a7d461675012f9bbe7a03ff2e WatchSource:0}: Error finding container c2ed5a264cfe779495e2ff575dc906bc0e54b86a7d461675012f9bbe7a03ff2e: Status 404 returned error can't find the container with id c2ed5a264cfe779495e2ff575dc906bc0e54b86a7d461675012f9bbe7a03ff2e
Sep 10 23:44:38.834556 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:38.834532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv" event={"ID":"9a1599a9-3e16-4923-9482-2155fdeebff9","Type":"ContainerStarted","Data":"c2ed5a264cfe779495e2ff575dc906bc0e54b86a7d461675012f9bbe7a03ff2e"}
Sep 10 23:44:43.862496 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:43.862464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv" event={"ID":"9a1599a9-3e16-4923-9482-2155fdeebff9","Type":"ContainerStarted","Data":"3207e4b57db852f8efccd205ceab1496aae396523b783e7391b0cce2849fc61c"}
Sep 10 23:44:43.862809 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:43.862535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:43.879545 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:43.879499    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv" podStartSLOduration=0.838523102 podStartE2EDuration="5.879486003s" totalImagesPullingTime="5.040962901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:44:43.878270821 +0000 UTC m=+495.709298375" watchObservedRunningTime="2026-09-10 23:44:43.879486003 +0000 UTC m=+495.710513536"
Sep 10 23:44:49.896054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:49.896020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sr9bv"
Sep 10 23:44:50.216969 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.216917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:50.458279 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.458246    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dlg8b"]
Sep 10 23:44:50.464344 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.464323    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dlg8b"
Sep 10 23:44:50.467817 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.467760    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rcb9d\""
Sep 10 23:44:50.490166 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.490144    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dlg8b"]
Sep 10 23:44:50.560443 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.560403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:44:50.609640 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.609613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjkt\" (UniqueName: \"kubernetes.io/projected/64c83083-89bf-4f78-9f52-1ab06d7562cb-kube-api-access-mzjkt\") pod \"authorino-557f876f69-dlg8b\" (UID: \"64c83083-89bf-4f78-9f52-1ab06d7562cb\") " pod="kuadrant-system/authorino-557f876f69-dlg8b"
Sep 10 23:44:50.625359 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.625332    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dlg8b"]
Sep 10 23:44:50.625553 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:44:50.625536    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mzjkt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-dlg8b" podUID="64c83083-89bf-4f78-9f52-1ab06d7562cb"
Sep 10 23:44:50.655309 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.655287    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dwlsf"]
Sep 10 23:44:50.659296 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.659281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.661737 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.661718    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 23:44:50.670079 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.670059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dwlsf"]
Sep 10 23:44:50.689586 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.689563    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dwlsf"]
Sep 10 23:44:50.689768 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:44:50.689752    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-f4n25 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-f4n25 tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-dwlsf" podUID="d2af621e-ecf5-4b06-9b65-75a5f7b3fef3"
Sep 10 23:44:50.710356 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.710329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjkt\" (UniqueName: \"kubernetes.io/projected/64c83083-89bf-4f78-9f52-1ab06d7562cb-kube-api-access-mzjkt\") pod \"authorino-557f876f69-dlg8b\" (UID: \"64c83083-89bf-4f78-9f52-1ab06d7562cb\") " pod="kuadrant-system/authorino-557f876f69-dlg8b"
Sep 10 23:44:50.721024 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.720977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjkt\" (UniqueName: \"kubernetes.io/projected/64c83083-89bf-4f78-9f52-1ab06d7562cb-kube-api-access-mzjkt\") pod \"authorino-557f876f69-dlg8b\" (UID: \"64c83083-89bf-4f78-9f52-1ab06d7562cb\") " pod="kuadrant-system/authorino-557f876f69-dlg8b"
Sep 10 23:44:50.721496 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.721476    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cd5655596-q25wn"]
Sep 10 23:44:50.725794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.725778    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:44:50.731710 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.731691    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd5655596-q25wn"]
Sep 10 23:44:50.811414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.811392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7a39608-41a5-4c0c-a4e2-254504c23b9b-tls-cert\") pod \"authorino-6cd5655596-q25wn\" (UID: \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\") " pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:44:50.811519 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.811418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzknx\" (UniqueName: \"kubernetes.io/projected/a7a39608-41a5-4c0c-a4e2-254504c23b9b-kube-api-access-dzknx\") pod \"authorino-6cd5655596-q25wn\" (UID: \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\") " pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:44:50.811519 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.811441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-tls-cert\") pod \"authorino-6d4dfc649f-dwlsf\" (UID: \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\") " pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.811636 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.811606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4n25\" (UniqueName: \"kubernetes.io/projected/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-kube-api-access-f4n25\") pod \"authorino-6d4dfc649f-dwlsf\" (UID: \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\") " pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.894243 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.894220    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.894334 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.894222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dlg8b"
Sep 10 23:44:50.901139 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.901122    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.904019 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.904005    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dlg8b"
Sep 10 23:44:50.912075 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.912049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7a39608-41a5-4c0c-a4e2-254504c23b9b-tls-cert\") pod \"authorino-6cd5655596-q25wn\" (UID: \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\") " pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:44:50.912155 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.912079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzknx\" (UniqueName: \"kubernetes.io/projected/a7a39608-41a5-4c0c-a4e2-254504c23b9b-kube-api-access-dzknx\") pod \"authorino-6cd5655596-q25wn\" (UID: \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\") " pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:44:50.912155 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.912100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-tls-cert\") pod \"authorino-6d4dfc649f-dwlsf\" (UID: \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\") " pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.912227 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.912162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4n25\" (UniqueName: \"kubernetes.io/projected/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-kube-api-access-f4n25\") pod \"authorino-6d4dfc649f-dwlsf\" (UID: \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\") " pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.914249 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.914226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7a39608-41a5-4c0c-a4e2-254504c23b9b-tls-cert\") pod \"authorino-6cd5655596-q25wn\" (UID: \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\") " pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:44:50.914536 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.914518    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-tls-cert\") pod \"authorino-6d4dfc649f-dwlsf\" (UID: \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\") " pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.919373 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.919350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4n25\" (UniqueName: \"kubernetes.io/projected/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-kube-api-access-f4n25\") pod \"authorino-6d4dfc649f-dwlsf\" (UID: \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\") " pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:50.919675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:50.919655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzknx\" (UniqueName: \"kubernetes.io/projected/a7a39608-41a5-4c0c-a4e2-254504c23b9b-kube-api-access-dzknx\") pod \"authorino-6cd5655596-q25wn\" (UID: \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\") " pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:44:51.012972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.012890    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-tls-cert\") pod \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\" (UID: \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\") "
Sep 10 23:44:51.012972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.012959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4n25\" (UniqueName: \"kubernetes.io/projected/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-kube-api-access-f4n25\") pod \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\" (UID: \"d2af621e-ecf5-4b06-9b65-75a5f7b3fef3\") "
Sep 10 23:44:51.013132 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.013050    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzjkt\" (UniqueName: \"kubernetes.io/projected/64c83083-89bf-4f78-9f52-1ab06d7562cb-kube-api-access-mzjkt\") pod \"64c83083-89bf-4f78-9f52-1ab06d7562cb\" (UID: \"64c83083-89bf-4f78-9f52-1ab06d7562cb\") "
Sep 10 23:44:51.014651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.014627    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-kube-api-access-f4n25" (OuterVolumeSpecName: "kube-api-access-f4n25") pod "d2af621e-ecf5-4b06-9b65-75a5f7b3fef3" (UID: "d2af621e-ecf5-4b06-9b65-75a5f7b3fef3"). InnerVolumeSpecName "kube-api-access-f4n25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:44:51.014795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.014702    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d2af621e-ecf5-4b06-9b65-75a5f7b3fef3" (UID: "d2af621e-ecf5-4b06-9b65-75a5f7b3fef3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:44:51.014795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.014787    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c83083-89bf-4f78-9f52-1ab06d7562cb-kube-api-access-mzjkt" (OuterVolumeSpecName: "kube-api-access-mzjkt") pod "64c83083-89bf-4f78-9f52-1ab06d7562cb" (UID: "64c83083-89bf-4f78-9f52-1ab06d7562cb"). InnerVolumeSpecName "kube-api-access-mzjkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:44:51.038080 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.038060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:44:51.114550 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.114518    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjkt\" (UniqueName: \"kubernetes.io/projected/64c83083-89bf-4f78-9f52-1ab06d7562cb-kube-api-access-mzjkt\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:44:51.114659 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.114557    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-tls-cert\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:44:51.114659 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.114575    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4n25\" (UniqueName: \"kubernetes.io/projected/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3-kube-api-access-f4n25\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:44:51.305308 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.305236    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd5655596-q25wn"]
Sep 10 23:44:51.308161 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:44:51.308131    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7a39608_41a5_4c0c_a4e2_254504c23b9b.slice/crio-7736185cfcd7da9b2e60644bce25f8f6231f645e766df526dace8589b6063383 WatchSource:0}: Error finding container 7736185cfcd7da9b2e60644bce25f8f6231f645e766df526dace8589b6063383: Status 404 returned error can't find the container with id 7736185cfcd7da9b2e60644bce25f8f6231f645e766df526dace8589b6063383
Sep 10 23:44:51.899681 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.899644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd5655596-q25wn" event={"ID":"a7a39608-41a5-4c0c-a4e2-254504c23b9b","Type":"ContainerStarted","Data":"7736185cfcd7da9b2e60644bce25f8f6231f645e766df526dace8589b6063383"}
Sep 10 23:44:51.899681 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.899680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dlg8b"
Sep 10 23:44:51.899913 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.899680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dwlsf"
Sep 10 23:44:51.931730 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.931699    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dwlsf"]
Sep 10 23:44:51.933325 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.933303    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dwlsf"]
Sep 10 23:44:51.953914 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.953893    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dlg8b"]
Sep 10 23:44:51.956204 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:51.956184    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dlg8b"]
Sep 10 23:44:52.654664 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:52.654634    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c83083-89bf-4f78-9f52-1ab06d7562cb" path="/var/lib/kubelet/pods/64c83083-89bf-4f78-9f52-1ab06d7562cb/volumes"
Sep 10 23:44:52.654886 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:44:52.654873    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2af621e-ecf5-4b06-9b65-75a5f7b3fef3" path="/var/lib/kubelet/pods/d2af621e-ecf5-4b06-9b65-75a5f7b3fef3/volumes"
Sep 10 23:45:09.976435 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:09.976400    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd5655596-q25wn" event={"ID":"a7a39608-41a5-4c0c-a4e2-254504c23b9b","Type":"ContainerStarted","Data":"a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75"}
Sep 10 23:45:09.993360 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:09.993318    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cd5655596-q25wn" podStartSLOduration=2.212558645 podStartE2EDuration="19.993303849s" totalImagesPullingTime="17.780745204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:45:09.99270931 +0000 UTC m=+521.823736870" watchObservedRunningTime="2026-09-10 23:45:09.993303849 +0000 UTC m=+521.824331381"
Sep 10 23:45:14.660477 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.660444    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:14.679337 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.679309    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58dc578786-rmp2j"]
Sep 10 23:45:14.682458 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.682442    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.685050 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.685031    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 23:45:14.686262 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.686217    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 23:45:14.686262 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.686238    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mmgzc\""
Sep 10 23:45:14.686262 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.686217    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 23:45:14.696294 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.696273    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58dc578786-rmp2j"]
Sep 10 23:45:14.757768 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.757745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:14.787509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.787488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgqg\" (UniqueName: \"kubernetes.io/projected/e51cd400-9e45-4257-b4e3-2a65735f05bb-kube-api-access-fwgqg\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.787606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.787528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-webhook-cert\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.787606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.787552    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-metrics-certs\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.846279 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.846254    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:14.850118 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.850092    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-t9bkz"]
Sep 10 23:45:14.853314 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.853300    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.863571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.863549    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-t9bkz"]
Sep 10 23:45:14.888606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.888586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-webhook-cert\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.888708 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.888611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9mj9\" (UniqueName: \"kubernetes.io/projected/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-kube-api-access-v9mj9\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.888708 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.888642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgqg\" (UniqueName: \"kubernetes.io/projected/e51cd400-9e45-4257-b4e3-2a65735f05bb-kube-api-access-fwgqg\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.888708 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.888675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-webhook-cert\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.888708 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.888703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-metrics-certs\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.888853 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.888737    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-metrics-certs\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.891026 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.891005    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-webhook-cert\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.891131 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.891111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-metrics-certs\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.898260 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.898239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgqg\" (UniqueName: \"kubernetes.io/projected/e51cd400-9e45-4257-b4e3-2a65735f05bb-kube-api-access-fwgqg\") pod \"maas-controller-58dc578786-rmp2j\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") " pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.922677 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.922625    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:14.990097 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.990070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-metrics-certs\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.990192 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.990110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-webhook-cert\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.990192 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.990135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9mj9\" (UniqueName: \"kubernetes.io/projected/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-kube-api-access-v9mj9\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.991975 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.991955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:14.992239 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.992223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-metrics-certs\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.992300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.992283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-webhook-cert\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:14.998293 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:14.998268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9mj9\" (UniqueName: \"kubernetes.io/projected/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-kube-api-access-v9mj9\") pod \"maas-controller-54dc99dcdb-t9bkz\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") " pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:15.163609 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:15.163579    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:15.260009 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:15.259980    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58dc578786-rmp2j"]
Sep 10 23:45:15.263331 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:45:15.263298    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51cd400_9e45_4257_b4e3_2a65735f05bb.slice/crio-030acb2b6e5fe83d1b314e5378596d4f39507d580eb779916bb72c05dd78c00b WatchSource:0}: Error finding container 030acb2b6e5fe83d1b314e5378596d4f39507d580eb779916bb72c05dd78c00b: Status 404 returned error can't find the container with id 030acb2b6e5fe83d1b314e5378596d4f39507d580eb779916bb72c05dd78c00b
Sep 10 23:45:15.433150 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:15.433071    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-t9bkz"]
Sep 10 23:45:15.435561 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:45:15.435536    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5aa089a_7620_4bb2_80fb_433c6f4a48f9.slice/crio-785291be5675a018071a254da286aa3499f8709f7c6572028856b6a418ce47ef WatchSource:0}: Error finding container 785291be5675a018071a254da286aa3499f8709f7c6572028856b6a418ce47ef: Status 404 returned error can't find the container with id 785291be5675a018071a254da286aa3499f8709f7c6572028856b6a418ce47ef
Sep 10 23:45:16.006231 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:16.006193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58dc578786-rmp2j" event={"ID":"e51cd400-9e45-4257-b4e3-2a65735f05bb","Type":"ContainerStarted","Data":"030acb2b6e5fe83d1b314e5378596d4f39507d580eb779916bb72c05dd78c00b"}
Sep 10 23:45:16.009672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:16.009639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz" event={"ID":"c5aa089a-7620-4bb2-80fb-433c6f4a48f9","Type":"ContainerStarted","Data":"785291be5675a018071a254da286aa3499f8709f7c6572028856b6a418ce47ef"}
Sep 10 23:45:18.147826 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:18.147787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:18.255125 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:18.255051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:19.033211 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:19.033172    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz" event={"ID":"c5aa089a-7620-4bb2-80fb-433c6f4a48f9","Type":"ContainerStarted","Data":"9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53"}
Sep 10 23:45:19.033451 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:19.033238    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:19.034558 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:19.034533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58dc578786-rmp2j" event={"ID":"e51cd400-9e45-4257-b4e3-2a65735f05bb","Type":"ContainerStarted","Data":"304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1"}
Sep 10 23:45:19.034670 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:19.034657    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:19.067541 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:19.067491    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz" podStartSLOduration=2.521544673 podStartE2EDuration="5.067473321s" totalImagesPullingTime="2.545928648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:45:19.06523933 +0000 UTC m=+530.896266883" watchObservedRunningTime="2026-09-10 23:45:19.067473321 +0000 UTC m=+530.898500849"
Sep 10 23:45:19.101070 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:19.101032    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58dc578786-rmp2j" podStartSLOduration=2.38955159 podStartE2EDuration="5.10102282s" totalImagesPullingTime="2.71147123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:45:19.098686677 +0000 UTC m=+530.929714210" watchObservedRunningTime="2026-09-10 23:45:19.10102282 +0000 UTC m=+530.932050385"
Sep 10 23:45:30.043334 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.043302    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 10 23:45:30.043694 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.043516    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:30.094919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.094888    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58dc578786-rmp2j"]
Sep 10 23:45:30.095181 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.095153    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58dc578786-rmp2j" podUID="e51cd400-9e45-4257-b4e3-2a65735f05bb" containerName="manager" containerID="cri-o://304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1" gracePeriod=30
Sep 10 23:45:30.336560 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.336538    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:30.412415 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.412385    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwgqg\" (UniqueName: \"kubernetes.io/projected/e51cd400-9e45-4257-b4e3-2a65735f05bb-kube-api-access-fwgqg\") pod \"e51cd400-9e45-4257-b4e3-2a65735f05bb\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") "
Sep 10 23:45:30.412552 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.412438    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-metrics-certs\") pod \"e51cd400-9e45-4257-b4e3-2a65735f05bb\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") "
Sep 10 23:45:30.412552 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.412480    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-webhook-cert\") pod \"e51cd400-9e45-4257-b4e3-2a65735f05bb\" (UID: \"e51cd400-9e45-4257-b4e3-2a65735f05bb\") "
Sep 10 23:45:30.414529 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.414504    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e51cd400-9e45-4257-b4e3-2a65735f05bb" (UID: "e51cd400-9e45-4257-b4e3-2a65735f05bb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:45:30.414645 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.414577    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e51cd400-9e45-4257-b4e3-2a65735f05bb" (UID: "e51cd400-9e45-4257-b4e3-2a65735f05bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:45:30.414645 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.414597    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51cd400-9e45-4257-b4e3-2a65735f05bb-kube-api-access-fwgqg" (OuterVolumeSpecName: "kube-api-access-fwgqg") pod "e51cd400-9e45-4257-b4e3-2a65735f05bb" (UID: "e51cd400-9e45-4257-b4e3-2a65735f05bb"). InnerVolumeSpecName "kube-api-access-fwgqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:45:30.513244 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.513219    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:45:30.513244 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.513242    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e51cd400-9e45-4257-b4e3-2a65735f05bb-webhook-cert\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:45:30.513374 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:30.513251    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwgqg\" (UniqueName: \"kubernetes.io/projected/e51cd400-9e45-4257-b4e3-2a65735f05bb-kube-api-access-fwgqg\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:45:31.091213 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.091180    2562 generic.go:356] "Generic (PLEG): container finished" podID="e51cd400-9e45-4257-b4e3-2a65735f05bb" containerID="304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1" exitCode=0
Sep 10 23:45:31.091560 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.091231    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58dc578786-rmp2j"
Sep 10 23:45:31.091560 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.091265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58dc578786-rmp2j" event={"ID":"e51cd400-9e45-4257-b4e3-2a65735f05bb","Type":"ContainerDied","Data":"304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1"}
Sep 10 23:45:31.091560 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.091315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58dc578786-rmp2j" event={"ID":"e51cd400-9e45-4257-b4e3-2a65735f05bb","Type":"ContainerDied","Data":"030acb2b6e5fe83d1b314e5378596d4f39507d580eb779916bb72c05dd78c00b"}
Sep 10 23:45:31.091560 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.091337    2562 scope.go:117] "RemoveContainer" containerID="304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1"
Sep 10 23:45:31.099952 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.099736    2562 scope.go:117] "RemoveContainer" containerID="304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1"
Sep 10 23:45:31.100050 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:45:31.100031    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1\": container with ID starting with 304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1 not found: ID does not exist" containerID="304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1"
Sep 10 23:45:31.100093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.100058    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1"} err="failed to get container status \"304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1\": rpc error: code = NotFound desc = could not find container \"304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1\": container with ID starting with 304d1dbc2a903ad143347d5b9499af1f7b80a750e2c7b380fe7cefed63bdbce1 not found: ID does not exist"
Sep 10 23:45:31.107192 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.107170    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58dc578786-rmp2j"]
Sep 10 23:45:31.110887 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:31.110866    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58dc578786-rmp2j"]
Sep 10 23:45:32.655209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:32.655178    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e51cd400-9e45-4257-b4e3-2a65735f05bb" path="/var/lib/kubelet/pods/e51cd400-9e45-4257-b4e3-2a65735f05bb/volumes"
Sep 10 23:45:48.767519 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:48.767484    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:48.877490 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:48.877459    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:50.331619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.331586    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:50.388531 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.388501    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"]
Sep 10 23:45:50.388849 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.388831    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51cd400-9e45-4257-b4e3-2a65735f05bb" containerName="manager"
Sep 10 23:45:50.388849 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.388847    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51cd400-9e45-4257-b4e3-2a65735f05bb" containerName="manager"
Sep 10 23:45:50.389040 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.388919    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e51cd400-9e45-4257-b4e3-2a65735f05bb" containerName="manager"
Sep 10 23:45:50.394652 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.394634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.397288 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.397265    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 23:45:50.397388 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.397266    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:45:50.404406 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.404378    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"]
Sep 10 23:45:50.411162 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.411140    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:50.466506 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.466481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w4p\" (UniqueName: \"kubernetes.io/projected/a2292ac2-da55-4a36-81cd-ffcf33659dac-kube-api-access-r6w4p\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.466599 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.466535    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.466599 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.466574    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2292ac2-da55-4a36-81cd-ffcf33659dac-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.466672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.466611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.567794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.567756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w4p\" (UniqueName: \"kubernetes.io/projected/a2292ac2-da55-4a36-81cd-ffcf33659dac-kube-api-access-r6w4p\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.567968 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.567824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.567968 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.567842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2292ac2-da55-4a36-81cd-ffcf33659dac-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.567968 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.567863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.568140 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:45:50.567988    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 10 23:45:50.568140 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:45:50.568036    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 10 23:45:50.568140 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:45:50.568052    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs podName:a2292ac2-da55-4a36-81cd-ffcf33659dac nodeName:}" failed. No retries permitted until 2026-09-10 23:45:51.06803596 +0000 UTC m=+562.899063471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs") pod "maas-api-8f6fd9d5d-blm78" (UID: "a2292ac2-da55-4a36-81cd-ffcf33659dac") : secret "maas-api-metrics-tls" not found
Sep 10 23:45:50.568140 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:45:50.568099    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls podName:a2292ac2-da55-4a36-81cd-ffcf33659dac nodeName:}" failed. No retries permitted until 2026-09-10 23:45:51.06808513 +0000 UTC m=+562.899112641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls") pod "maas-api-8f6fd9d5d-blm78" (UID: "a2292ac2-da55-4a36-81cd-ffcf33659dac") : secret "maas-api-serving-cert" not found
Sep 10 23:45:50.568463 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.568446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2292ac2-da55-4a36-81cd-ffcf33659dac-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.576258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.576232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w4p\" (UniqueName: \"kubernetes.io/projected/a2292ac2-da55-4a36-81cd-ffcf33659dac-kube-api-access-r6w4p\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:50.639865 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.639800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:50.737995 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.737958    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:50.837356 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.837322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:50.907327 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.907250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:45:50.933567 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.933537    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"]
Sep 10 23:45:50.937529 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.937510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:50.940163 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.940142    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 23:45:50.940273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.940233    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-r8rth\""
Sep 10 23:45:50.951604 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:50.951583    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"]
Sep 10 23:45:51.072476 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.072436    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8srq\" (UniqueName: \"kubernetes.io/projected/eefba04a-fca1-4281-9427-8635b54bd72a-kube-api-access-k8srq\") pod \"payload-processing-5b54b7fbb7-5hmzr\" (UID: \"eefba04a-fca1-4281-9427-8635b54bd72a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:51.072625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.072517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefba04a-fca1-4281-9427-8635b54bd72a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-5hmzr\" (UID: \"eefba04a-fca1-4281-9427-8635b54bd72a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:51.072625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.072609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:51.072704 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.072639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:51.075304 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.075272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:51.075661 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.075635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs\") pod \"maas-api-8f6fd9d5d-blm78\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:51.173100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.173019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8srq\" (UniqueName: \"kubernetes.io/projected/eefba04a-fca1-4281-9427-8635b54bd72a-kube-api-access-k8srq\") pod \"payload-processing-5b54b7fbb7-5hmzr\" (UID: \"eefba04a-fca1-4281-9427-8635b54bd72a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:51.173100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.173064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefba04a-fca1-4281-9427-8635b54bd72a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-5hmzr\" (UID: \"eefba04a-fca1-4281-9427-8635b54bd72a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:51.173796 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.173765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefba04a-fca1-4281-9427-8635b54bd72a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-5hmzr\" (UID: \"eefba04a-fca1-4281-9427-8635b54bd72a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:51.182100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.182067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8srq\" (UniqueName: \"kubernetes.io/projected/eefba04a-fca1-4281-9427-8635b54bd72a-kube-api-access-k8srq\") pod \"payload-processing-5b54b7fbb7-5hmzr\" (UID: \"eefba04a-fca1-4281-9427-8635b54bd72a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:51.247496 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.247456    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:51.306276 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.306248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:51.521608 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.521583    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"]
Sep 10 23:45:51.523968 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:45:51.523945    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefba04a_fca1_4281_9427_8635b54bd72a.slice/crio-6a412c07409ec0a79bd7d6f18a2707f71f0e98837bca538e52f7fbcd7c9ab2eb WatchSource:0}: Error finding container 6a412c07409ec0a79bd7d6f18a2707f71f0e98837bca538e52f7fbcd7c9ab2eb: Status 404 returned error can't find the container with id 6a412c07409ec0a79bd7d6f18a2707f71f0e98837bca538e52f7fbcd7c9ab2eb
Sep 10 23:45:51.586215 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:51.586187    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"]
Sep 10 23:45:51.588819 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:45:51.588794    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2292ac2_da55_4a36_81cd_ffcf33659dac.slice/crio-43809159daad411fea5b2c65e076f87457bdf2ab9f5c614b90efc0634bf5eb6e WatchSource:0}: Error finding container 43809159daad411fea5b2c65e076f87457bdf2ab9f5c614b90efc0634bf5eb6e: Status 404 returned error can't find the container with id 43809159daad411fea5b2c65e076f87457bdf2ab9f5c614b90efc0634bf5eb6e
Sep 10 23:45:52.187387 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:52.187351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78" event={"ID":"a2292ac2-da55-4a36-81cd-ffcf33659dac","Type":"ContainerStarted","Data":"43809159daad411fea5b2c65e076f87457bdf2ab9f5c614b90efc0634bf5eb6e"}
Sep 10 23:45:52.189087 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:52.189048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr" event={"ID":"eefba04a-fca1-4281-9427-8635b54bd72a","Type":"ContainerStarted","Data":"6a412c07409ec0a79bd7d6f18a2707f71f0e98837bca538e52f7fbcd7c9ab2eb"}
Sep 10 23:45:55.205725 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:55.205689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78" event={"ID":"a2292ac2-da55-4a36-81cd-ffcf33659dac","Type":"ContainerStarted","Data":"307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f"}
Sep 10 23:45:55.206207 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:55.205838    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:45:55.207241 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:55.207220    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr" event={"ID":"eefba04a-fca1-4281-9427-8635b54bd72a","Type":"ContainerStarted","Data":"46468d4930e24d23ef847cdd4ccd14f279b545e275d2ab9aec7fdc08c007b37a"}
Sep 10 23:45:55.207352 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:55.207337    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:45:55.222294 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:55.222167    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78" podStartSLOduration=2.587492201 podStartE2EDuration="5.222151625s" totalImagesPullingTime="2.634659424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:45:55.221962999 +0000 UTC m=+567.052990551" watchObservedRunningTime="2026-09-10 23:45:55.222151625 +0000 UTC m=+567.053179159"
Sep 10 23:45:55.238218 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:45:55.238174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr" podStartSLOduration=2.491739057 podStartE2EDuration="5.238163076s" totalImagesPullingTime="2.746424019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:45:55.237821752 +0000 UTC m=+567.068849295" watchObservedRunningTime="2026-09-10 23:45:55.238163076 +0000 UTC m=+567.069190609"
Sep 10 23:46:01.216963 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:01.216916    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 10 23:46:02.298296 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:02.298265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:02.398541 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:02.398495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:06.213038 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:06.213005    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-5hmzr"
Sep 10 23:46:15.024256 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:15.024223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:20.433989 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:20.433952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:20.544998 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:20.544969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:28.566093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:28.566032    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:46:28.568249 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:28.568210    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:46:29.641933 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:29.641889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:47.231792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.231759    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"]
Sep 10 23:46:47.235411 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.235392    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.238078 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.238050    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 23:46:47.238181 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.238165    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rmfcp\""
Sep 10 23:46:47.238876 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.238851    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 23:46:47.239019 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.238999    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 23:46:47.244316 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.244287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"]
Sep 10 23:46:47.269693 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.269670    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:47.304838 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.304812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.305008 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.304852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.305008 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.304881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvz8z\" (UniqueName: \"kubernetes.io/projected/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-kube-api-access-tvz8z\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.305008 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.304912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.305008 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.304997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.305191 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.305045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.405497 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.405625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.405625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405530    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvz8z\" (UniqueName: \"kubernetes.io/projected/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-kube-api-access-tvz8z\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.405625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.405625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.405625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.405947 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.406020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.406020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.405973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.407780 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.407763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.408209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.408188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.413157 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.413138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvz8z\" (UniqueName: \"kubernetes.io/projected/77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed-kube-api-access-tvz8z\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4\" (UID: \"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.547899 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.547842    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:47.814800 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.814777    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"]
Sep 10 23:46:47.816691 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:46:47.816664    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b2f51e_658c_4037_8dd4_6b3e0eb2b6ed.slice/crio-50917dee2e15131186d9fa91eacbdf33de000abfd0688ae5909c90ac0992cb78 WatchSource:0}: Error finding container 50917dee2e15131186d9fa91eacbdf33de000abfd0688ae5909c90ac0992cb78: Status 404 returned error can't find the container with id 50917dee2e15131186d9fa91eacbdf33de000abfd0688ae5909c90ac0992cb78
Sep 10 23:46:47.817968 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:47.817946    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:46:48.443020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:48.442967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4" event={"ID":"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed","Type":"ContainerStarted","Data":"50917dee2e15131186d9fa91eacbdf33de000abfd0688ae5909c90ac0992cb78"}
Sep 10 23:46:49.450075 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:49.450039    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4" event={"ID":"77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed","Type":"ContainerStarted","Data":"f46b353312fbe19040ae72ee6e84d915bac00846b6d2888481c7a99a6dab9d7b"}
Sep 10 23:46:49.469654 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:49.469608    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4" podStartSLOduration=1.3167551180000001 podStartE2EDuration="2.469592439s" totalImagesPullingTime="1.152837321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:46:49.468070321 +0000 UTC m=+621.299097873" watchObservedRunningTime="2026-09-10 23:46:49.469592439 +0000 UTC m=+621.300619971"
Sep 10 23:46:54.859552 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:54.859480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:46:57.548836 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:57.548801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:57.549270 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:57.548865    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:57.561162 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:57.561140    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:46:58.501967 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:46:58.501922    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4"
Sep 10 23:47:23.829850 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.829817    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"]
Sep 10 23:47:23.833990 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.833973    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:23.836446 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.836420    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 23:47:23.840500 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.840475    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"]
Sep 10 23:47:23.857136 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.857113    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:23.991278 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.991251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:23.991443 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.991302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f647124-63d3-40d9-9634-b97bcbe915a1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:23.991443 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.991326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:23.991443 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.991343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:23.991443 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.991359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvg8\" (UniqueName: \"kubernetes.io/projected/5f647124-63d3-40d9-9634-b97bcbe915a1-kube-api-access-wcvg8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:23.991443 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:23.991389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092226 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092226 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092226 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvg8\" (UniqueName: \"kubernetes.io/projected/5f647124-63d3-40d9-9634-b97bcbe915a1-kube-api-access-wcvg8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092484 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092484 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092484 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f647124-63d3-40d9-9634-b97bcbe915a1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092618 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092618 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.092685 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.092636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.094567 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.094539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f647124-63d3-40d9-9634-b97bcbe915a1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.094804 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.094789    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f647124-63d3-40d9-9634-b97bcbe915a1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.099749 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.099723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvg8\" (UniqueName: \"kubernetes.io/projected/5f647124-63d3-40d9-9634-b97bcbe915a1-kube-api-access-wcvg8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b\" (UID: \"5f647124-63d3-40d9-9634-b97bcbe915a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.144618 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.144594    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:24.414960 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.414816    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"]
Sep 10 23:47:24.417849 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:47:24.417822    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f647124_63d3_40d9_9634_b97bcbe915a1.slice/crio-f0e06e9c5d8cb3bdf03c5962b0108c15be51c46e1f69f9618b5d824f12562737 WatchSource:0}: Error finding container f0e06e9c5d8cb3bdf03c5962b0108c15be51c46e1f69f9618b5d824f12562737: Status 404 returned error can't find the container with id f0e06e9c5d8cb3bdf03c5962b0108c15be51c46e1f69f9618b5d824f12562737
Sep 10 23:47:24.604512 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:24.604488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b" event={"ID":"5f647124-63d3-40d9-9634-b97bcbe915a1","Type":"ContainerStarted","Data":"f0e06e9c5d8cb3bdf03c5962b0108c15be51c46e1f69f9618b5d824f12562737"}
Sep 10 23:47:25.611091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:25.611050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b" event={"ID":"5f647124-63d3-40d9-9634-b97bcbe915a1","Type":"ContainerStarted","Data":"9fff0788812cfd8f7ea35d9b20bdd17d2a4d54f67eae53e958f18759e9abcc26"}
Sep 10 23:47:25.635249 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:25.635188    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b" podStartSLOduration=2.460713526 podStartE2EDuration="2.635168402s" totalImagesPullingTime="174.454876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:47:25.633074767 +0000 UTC m=+657.464102317" watchObservedRunningTime="2026-09-10 23:47:25.635168402 +0000 UTC m=+657.466195936"
Sep 10 23:47:29.849164 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:29.849125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:34.145348 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:34.145315    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:34.145730 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:34.145495    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:34.158061 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:34.158036    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:34.664828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:34.664798    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b"
Sep 10 23:47:46.066329 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:46.066288    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:46.166382 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:46.166353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:47:50.830076 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:47:50.830047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:48:38.766694 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:48:38.766615    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:02.738637 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:02.738602    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cd5655596-q25wn"]
Sep 10 23:49:02.739132 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:02.738786    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cd5655596-q25wn" podUID="a7a39608-41a5-4c0c-a4e2-254504c23b9b" containerName="authorino" containerID="cri-o://a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75" gracePeriod=30
Sep 10 23:49:02.982172 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:02.982148    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:49:03.034545 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.034476    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7a39608-41a5-4c0c-a4e2-254504c23b9b-tls-cert\") pod \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\" (UID: \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\") "
Sep 10 23:49:03.034684 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.034588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzknx\" (UniqueName: \"kubernetes.io/projected/a7a39608-41a5-4c0c-a4e2-254504c23b9b-kube-api-access-dzknx\") pod \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\" (UID: \"a7a39608-41a5-4c0c-a4e2-254504c23b9b\") "
Sep 10 23:49:03.036547 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.036519    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a39608-41a5-4c0c-a4e2-254504c23b9b-kube-api-access-dzknx" (OuterVolumeSpecName: "kube-api-access-dzknx") pod "a7a39608-41a5-4c0c-a4e2-254504c23b9b" (UID: "a7a39608-41a5-4c0c-a4e2-254504c23b9b"). InnerVolumeSpecName "kube-api-access-dzknx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:49:03.044509 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.044484    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a39608-41a5-4c0c-a4e2-254504c23b9b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a7a39608-41a5-4c0c-a4e2-254504c23b9b" (UID: "a7a39608-41a5-4c0c-a4e2-254504c23b9b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:49:03.051878 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.051856    2562 generic.go:356] "Generic (PLEG): container finished" podID="a7a39608-41a5-4c0c-a4e2-254504c23b9b" containerID="a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75" exitCode=0
Sep 10 23:49:03.051999 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.051911    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd5655596-q25wn"
Sep 10 23:49:03.051999 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.051951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd5655596-q25wn" event={"ID":"a7a39608-41a5-4c0c-a4e2-254504c23b9b","Type":"ContainerDied","Data":"a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75"}
Sep 10 23:49:03.051999 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.051991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd5655596-q25wn" event={"ID":"a7a39608-41a5-4c0c-a4e2-254504c23b9b","Type":"ContainerDied","Data":"7736185cfcd7da9b2e60644bce25f8f6231f645e766df526dace8589b6063383"}
Sep 10 23:49:03.052116 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.052009    2562 scope.go:117] "RemoveContainer" containerID="a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75"
Sep 10 23:49:03.060202 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.060185    2562 scope.go:117] "RemoveContainer" containerID="a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75"
Sep 10 23:49:03.060482 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:03.060461    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75\": container with ID starting with a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75 not found: ID does not exist" containerID="a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75"
Sep 10 23:49:03.060571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.060485    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75"} err="failed to get container status \"a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75\": rpc error: code = NotFound desc = could not find container \"a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75\": container with ID starting with a10aba1c7c8e1b51158bfe3e7fa201e17b9b05009834dd44e8eef6a33585af75 not found: ID does not exist"
Sep 10 23:49:03.075220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.075195    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cd5655596-q25wn"]
Sep 10 23:49:03.077805 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.077784    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cd5655596-q25wn"]
Sep 10 23:49:03.135684 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.135657    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzknx\" (UniqueName: \"kubernetes.io/projected/a7a39608-41a5-4c0c-a4e2-254504c23b9b-kube-api-access-dzknx\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:49:03.135684 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.135681    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7a39608-41a5-4c0c-a4e2-254504c23b9b-tls-cert\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:49:03.251820 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.251788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:03.350671 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:03.350564    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:04.357792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:04.357761    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:04.456211 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:04.456184    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:04.655788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:04.655712    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a39608-41a5-4c0c-a4e2-254504c23b9b" path="/var/lib/kubelet/pods/a7a39608-41a5-4c0c-a4e2-254504c23b9b/volumes"
Sep 10 23:49:19.602250 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:19.602194    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:19.707332 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:19.707298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:19.939685 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:19.939653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:20.173722 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:20.173655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:20.947181 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:20.947150    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:21.045329 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.045291    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:21.386846 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.386767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:21.527885 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.527850    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-6fe72b"]
Sep 10 23:49:21.528202 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.528189    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a39608-41a5-4c0c-a4e2-254504c23b9b" containerName="authorino"
Sep 10 23:49:21.528254 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.528204    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a39608-41a5-4c0c-a4e2-254504c23b9b" containerName="authorino"
Sep 10 23:49:21.528288 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.528271    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a39608-41a5-4c0c-a4e2-254504c23b9b" containerName="authorino"
Sep 10 23:49:21.532151 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.532125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:21.537559 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.537540    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nfkgj\""
Sep 10 23:49:21.543789 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.543768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-6fe72b"]
Sep 10 23:49:21.573492 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.573462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7skz\" (UniqueName: \"kubernetes.io/projected/bef330ea-e778-45ad-95b1-3c8a43855c1d-kube-api-access-q7skz\") pod \"test-curl-7701-6fe72b\" (UID: \"bef330ea-e778-45ad-95b1-3c8a43855c1d\") " pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:21.674031 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.673972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7skz\" (UniqueName: \"kubernetes.io/projected/bef330ea-e778-45ad-95b1-3c8a43855c1d-kube-api-access-q7skz\") pod \"test-curl-7701-6fe72b\" (UID: \"bef330ea-e778-45ad-95b1-3c8a43855c1d\") " pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:21.684761 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.684745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7skz\" (UniqueName: \"kubernetes.io/projected/bef330ea-e778-45ad-95b1-3c8a43855c1d-kube-api-access-q7skz\") pod \"test-curl-7701-6fe72b\" (UID: \"bef330ea-e778-45ad-95b1-3c8a43855c1d\") " pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:21.841222 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:21.841190    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:22.108205 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:22.108185    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-6fe72b"]
Sep 10 23:49:22.110561 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:49:22.110534    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbef330ea_e778_45ad_95b1_3c8a43855c1d.slice/crio-0f3a6d26c6a0ca976343fc31d3ed68a42177efdbbc70aefdab0ae8257c54dd60 WatchSource:0}: Error finding container 0f3a6d26c6a0ca976343fc31d3ed68a42177efdbbc70aefdab0ae8257c54dd60: Status 404 returned error can't find the container with id 0f3a6d26c6a0ca976343fc31d3ed68a42177efdbbc70aefdab0ae8257c54dd60
Sep 10 23:49:22.140615 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:22.140592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-6fe72b" event={"ID":"bef330ea-e778-45ad-95b1-3c8a43855c1d","Type":"ContainerStarted","Data":"0f3a6d26c6a0ca976343fc31d3ed68a42177efdbbc70aefdab0ae8257c54dd60"}
Sep 10 23:49:23.146602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.146522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-6fe72b" event={"ID":"bef330ea-e778-45ad-95b1-3c8a43855c1d","Type":"ContainerStarted","Data":"8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477"}
Sep 10 23:49:23.161297 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.161233    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7701-6fe72b" podStartSLOduration=1.419584256 podStartE2EDuration="2.161207213s" totalImagesPullingTime="741.622957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:23.159716646 +0000 UTC m=+774.990744178" watchObservedRunningTime="2026-09-10 23:49:23.161207213 +0000 UTC m=+774.992234748"
Sep 10 23:49:23.249090 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.249060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:23.346248 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.346219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:23.725640 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.725606    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-6fe72b"]
Sep 10 23:49:23.729799 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.729772    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-6fe72b"]
Sep 10 23:49:23.843939 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.843848    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-dcd12a"]
Sep 10 23:49:23.844250 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.844237    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef330ea-e778-45ad-95b1-3c8a43855c1d" containerName="test-curl-7701-6fe72b"
Sep 10 23:49:23.844295 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.844252    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef330ea-e778-45ad-95b1-3c8a43855c1d" containerName="test-curl-7701-6fe72b"
Sep 10 23:49:23.844328 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.844312    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef330ea-e778-45ad-95b1-3c8a43855c1d" containerName="test-curl-7701-6fe72b"
Sep 10 23:49:23.847288 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.847271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:23.854325 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.854281    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-dcd12a"]
Sep 10 23:49:23.892440 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.892418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79bg\" (UniqueName: \"kubernetes.io/projected/0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6-kube-api-access-d79bg\") pod \"test-curl-7701-dcd12a\" (UID: \"0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6\") " pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:23.993569 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:23.993543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79bg\" (UniqueName: \"kubernetes.io/projected/0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6-kube-api-access-d79bg\") pod \"test-curl-7701-dcd12a\" (UID: \"0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6\") " pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:24.001969 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.001948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79bg\" (UniqueName: \"kubernetes.io/projected/0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6-kube-api-access-d79bg\") pod \"test-curl-7701-dcd12a\" (UID: \"0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6\") " pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:24.152046 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:24.151964    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7701-6fe72b" container="test-curl-7701-6fe72b"
Sep 10 23:49:24.157289 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.157261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:24.367728 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.367695    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"]
Sep 10 23:49:24.372324 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.372297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.374721 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.374671    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1fe290-serving-cert\""
Sep 10 23:49:24.374877 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.374831    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9tsnw\""
Sep 10 23:49:24.382284 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.382226    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"]
Sep 10 23:49:24.419136 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.419076    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:24.429252 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.429217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-dcd12a"]
Sep 10 23:49:24.431256 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:49:24.431234    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0efdfc16_3f35_4546_bd84_6a2d7b1b1ce6.slice/crio-a9cf2a44deb302edfcb55acdb27792ca43bb1df2e847a24c64f3f30bf343980c WatchSource:0}: Error finding container a9cf2a44deb302edfcb55acdb27792ca43bb1df2e847a24c64f3f30bf343980c: Status 404 returned error can't find the container with id a9cf2a44deb302edfcb55acdb27792ca43bb1df2e847a24c64f3f30bf343980c
Sep 10 23:49:24.498107 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.498081    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hmqb\" (UniqueName: \"kubernetes.io/projected/8a698155-f53a-427c-989f-abdaab2357d5-kube-api-access-9hmqb\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.498203 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.498121    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698155-f53a-427c-989f-abdaab2357d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.498203 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.498180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.498328 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.498239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-metrics-certs\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.518304 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.518275    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:24.599617 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.599594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-metrics-certs\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.599734 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.599639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hmqb\" (UniqueName: \"kubernetes.io/projected/8a698155-f53a-427c-989f-abdaab2357d5-kube-api-access-9hmqb\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.599734 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.599665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698155-f53a-427c-989f-abdaab2357d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.599734 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.599692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.600378 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.600354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698155-f53a-427c-989f-abdaab2357d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.601908 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.601890    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.602002 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.601958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-metrics-certs\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.607146 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.607120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hmqb\" (UniqueName: \"kubernetes.io/projected/8a698155-f53a-427c-989f-abdaab2357d5-kube-api-access-9hmqb\") pod \"maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.686658 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.686637    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:24.967435 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:24.967403    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"]
Sep 10 23:49:24.969384 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:49:24.969354    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a698155_f53a_427c_989f_abdaab2357d5.slice/crio-b24122c06406f0bb8ccfc79fc688fbc7e3e2c38cfb25c3ad94335c6c19f26aa5 WatchSource:0}: Error finding container b24122c06406f0bb8ccfc79fc688fbc7e3e2c38cfb25c3ad94335c6c19f26aa5: Status 404 returned error can't find the container with id b24122c06406f0bb8ccfc79fc688fbc7e3e2c38cfb25c3ad94335c6c19f26aa5
Sep 10 23:49:25.004711 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.004688    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:25.101111 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.101080    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:25.138237 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.138201    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"]
Sep 10 23:49:25.142749 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.142732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:25.146388 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.146363    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1fe290\""
Sep 10 23:49:25.146492 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.146374    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1fe290-dockercfg-z46jc\""
Sep 10 23:49:25.156992 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.156966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw" event={"ID":"8a698155-f53a-427c-989f-abdaab2357d5","Type":"ContainerStarted","Data":"b24122c06406f0bb8ccfc79fc688fbc7e3e2c38cfb25c3ad94335c6c19f26aa5"}
Sep 10 23:49:25.158326 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.158307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-dcd12a" event={"ID":"0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6","Type":"ContainerStarted","Data":"09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d"}
Sep 10 23:49:25.158425 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.158341    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-dcd12a" event={"ID":"0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6","Type":"ContainerStarted","Data":"a9cf2a44deb302edfcb55acdb27792ca43bb1df2e847a24c64f3f30bf343980c"}
Sep 10 23:49:25.158425 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.158380    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7701-6fe72b" podUID="bef330ea-e778-45ad-95b1-3c8a43855c1d" containerName="test-curl-7701-6fe72b" containerID="cri-o://8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477" gracePeriod=2
Sep 10 23:49:25.161537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.161518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"]
Sep 10 23:49:25.190365 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.190330    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7701-dcd12a" podStartSLOduration=2.190318513 podStartE2EDuration="2.190318513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:25.188267816 +0000 UTC m=+777.019295350" watchObservedRunningTime="2026-09-10 23:49:25.190318513 +0000 UTC m=+777.021346046"
Sep 10 23:49:25.206072 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.206051    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4bd004-efd1-471c-9e1f-74580dbe5a37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm\" (UID: \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:25.206163 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.206111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvc5b\" (UniqueName: \"kubernetes.io/projected/0f4bd004-efd1-471c-9e1f-74580dbe5a37-kube-api-access-wvc5b\") pod \"payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm\" (UID: \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:25.307486 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.307402    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvc5b\" (UniqueName: \"kubernetes.io/projected/0f4bd004-efd1-471c-9e1f-74580dbe5a37-kube-api-access-wvc5b\") pod \"payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm\" (UID: \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:25.307624 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.307505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4bd004-efd1-471c-9e1f-74580dbe5a37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm\" (UID: \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:25.308309 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.308287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4bd004-efd1-471c-9e1f-74580dbe5a37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm\" (UID: \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:25.315539 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.315514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvc5b\" (UniqueName: \"kubernetes.io/projected/0f4bd004-efd1-471c-9e1f-74580dbe5a37-kube-api-access-wvc5b\") pod \"payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm\" (UID: \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:25.453215 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.453189    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:25.728358 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.728336    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"]
Sep 10 23:49:25.730205 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:49:25.730177    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4bd004_efd1_471c_9e1f_74580dbe5a37.slice/crio-371a46a682ccb6e175ea602d2205380fa86303f1e646050531e85d04500a3d1f WatchSource:0}: Error finding container 371a46a682ccb6e175ea602d2205380fa86303f1e646050531e85d04500a3d1f: Status 404 returned error can't find the container with id 371a46a682ccb6e175ea602d2205380fa86303f1e646050531e85d04500a3d1f
Sep 10 23:49:25.765575 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.765548    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-dcd12a"]
Sep 10 23:49:25.771824 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:25.771755    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-dcd12a"]
Sep 10 23:49:26.164518 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:26.164423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm" event={"ID":"0f4bd004-efd1-471c-9e1f-74580dbe5a37","Type":"ContainerStarted","Data":"af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659"}
Sep 10 23:49:26.164896 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:26.164524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm" event={"ID":"0f4bd004-efd1-471c-9e1f-74580dbe5a37","Type":"ContainerStarted","Data":"371a46a682ccb6e175ea602d2205380fa86303f1e646050531e85d04500a3d1f"}
Sep 10 23:49:26.164896 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:26.164556    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:26.166060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:26.166029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw" event={"ID":"8a698155-f53a-427c-989f-abdaab2357d5","Type":"ContainerStarted","Data":"a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0"}
Sep 10 23:49:26.166175 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:26.166159    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7701-dcd12a" container="test-curl-7701-dcd12a"
Sep 10 23:49:26.183445 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:26.183395    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm" podStartSLOduration=1.1833789239999999 podStartE2EDuration="1.183378924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:26.181840039 +0000 UTC m=+778.012867590" watchObservedRunningTime="2026-09-10 23:49:26.183378924 +0000 UTC m=+778.014406457"
Sep 10 23:49:26.199891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:26.199847    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw" podStartSLOduration=1.7853480240000001 podStartE2EDuration="2.199833046s" totalImagesPullingTime="414.485022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:26.199397755 +0000 UTC m=+778.030425308" watchObservedRunningTime="2026-09-10 23:49:26.199833046 +0000 UTC m=+778.030860578"
Sep 10 23:49:26.259513 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:26.259485    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:26.553278 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:26.553247    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:27.170487 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:27.170451    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7701-dcd12a" podUID="0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" containerName="test-curl-7701-dcd12a" containerID="cri-o://09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d" gracePeriod=2
Sep 10 23:49:27.170852 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:27.170793    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:27.340186 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:27.340165    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:27.426614 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:27.426561    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7skz\" (UniqueName: \"kubernetes.io/projected/bef330ea-e778-45ad-95b1-3c8a43855c1d-kube-api-access-q7skz\") pod \"bef330ea-e778-45ad-95b1-3c8a43855c1d\" (UID: \"bef330ea-e778-45ad-95b1-3c8a43855c1d\") "
Sep 10 23:49:27.428485 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:27.428463    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef330ea-e778-45ad-95b1-3c8a43855c1d-kube-api-access-q7skz" (OuterVolumeSpecName: "kube-api-access-q7skz") pod "bef330ea-e778-45ad-95b1-3c8a43855c1d" (UID: "bef330ea-e778-45ad-95b1-3c8a43855c1d"). InnerVolumeSpecName "kube-api-access-q7skz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:49:27.488381 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:27.488354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:27.527671 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:27.527648    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7skz\" (UniqueName: \"kubernetes.io/projected/bef330ea-e778-45ad-95b1-3c8a43855c1d-kube-api-access-q7skz\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:49:28.054237 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.054207    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:28.152810 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.152782    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:28.175963 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.175938    2562 generic.go:356] "Generic (PLEG): container finished" podID="bef330ea-e778-45ad-95b1-3c8a43855c1d" containerID="8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477" exitCode=137
Sep 10 23:49:28.176318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.175977    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:28.176318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.176036    2562 scope.go:117] "RemoveContainer" containerID="8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477"
Sep 10 23:49:28.185406 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.185388    2562 scope.go:117] "RemoveContainer" containerID="8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477"
Sep 10 23:49:28.185661 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:28.185645    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477\": container with ID starting with 8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477 not found: ID does not exist" containerID="8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477"
Sep 10 23:49:28.185719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.185671    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477"} err="failed to get container status \"8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477\": rpc error: code = NotFound desc = could not find container \"8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477\": container with ID starting with 8ac83bbaa013f910da87d1368d24a3ddf1f2b18c7d58d68843b6518ab6d1d477 not found: ID does not exist"
Sep 10 23:49:28.187078 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:28.187054    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-6fe72b\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="bef330ea-e778-45ad-95b1-3c8a43855c1d" pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:28.273667 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.273640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:28.569437 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.569405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:28.653518 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:28.653477    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-6fe72b\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="bef330ea-e778-45ad-95b1-3c8a43855c1d" pod="opendatahub/test-curl-7701-6fe72b"
Sep 10 23:49:28.656174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.656150    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bef330ea-e778-45ad-95b1-3c8a43855c1d" path="/var/lib/kubelet/pods/bef330ea-e778-45ad-95b1-3c8a43855c1d/volumes"
Sep 10 23:49:28.689361 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:28.689331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:29.310302 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:29.310261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:29.335248 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:29.335227    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:29.398292 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:29.398265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:29.443404 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:29.443379    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d79bg\" (UniqueName: \"kubernetes.io/projected/0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6-kube-api-access-d79bg\") pod \"0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6\" (UID: \"0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6\") "
Sep 10 23:49:29.445333 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:29.445315    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6-kube-api-access-d79bg" (OuterVolumeSpecName: "kube-api-access-d79bg") pod "0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" (UID: "0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6"). InnerVolumeSpecName "kube-api-access-d79bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:49:29.544886 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:29.544865    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79bg\" (UniqueName: \"kubernetes.io/projected/0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6-kube-api-access-d79bg\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:49:30.187620 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.187588    2562 generic.go:356] "Generic (PLEG): container finished" podID="0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" containerID="09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d" exitCode=137
Sep 10 23:49:30.187830 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.187634    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:30.187830 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.187687    2562 scope.go:117] "RemoveContainer" containerID="09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d"
Sep 10 23:49:30.196630 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.196432    2562 scope.go:117] "RemoveContainer" containerID="09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d"
Sep 10 23:49:30.196776 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:30.196753    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d\": container with ID starting with 09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d not found: ID does not exist" containerID="09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d"
Sep 10 23:49:30.196827 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.196788    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d"} err="failed to get container status \"09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d\": rpc error: code = NotFound desc = could not find container \"09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d\": container with ID starting with 09f7d46be4b44e1c73063a85865d97625945c3670741d76dc90906953181197d not found: ID does not exist"
Sep 10 23:49:30.198547 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:30.198522    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-dcd12a\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:30.472817 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.472738    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"]
Sep 10 23:49:30.473305 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.473267    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" containerName="test-curl-7701-dcd12a"
Sep 10 23:49:30.473305 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.473288    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" containerName="test-curl-7701-dcd12a"
Sep 10 23:49:30.473419 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.473408    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" containerName="test-curl-7701-dcd12a"
Sep 10 23:49:30.477145 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.477124    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.479543 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:30.479507    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-dcd12a\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" pod="opendatahub/test-curl-7701-dcd12a"
Sep 10 23:49:30.479675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.479662    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-74f1c1-serving-cert\""
Sep 10 23:49:30.504717 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.504697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"]
Sep 10 23:49:30.539058 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.539026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:30.552203 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.552183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-metrics-certs\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.552300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.552219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lph77\" (UniqueName: \"kubernetes.io/projected/13c51a1d-d12a-48da-a6ef-c88f5e897c05-kube-api-access-lph77\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.552300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.552241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c51a1d-d12a-48da-a6ef-c88f5e897c05-openshift-service-ca\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.552388 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.552349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.636059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.636028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:30.652726 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.652699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph77\" (UniqueName: \"kubernetes.io/projected/13c51a1d-d12a-48da-a6ef-c88f5e897c05-kube-api-access-lph77\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.652841 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.652740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c51a1d-d12a-48da-a6ef-c88f5e897c05-openshift-service-ca\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.652841 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.652816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.652965 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.652886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-metrics-certs\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.652965 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:30.652952    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-serving-cert: secret "maas-api-e2e-autores-74f1c1-serving-cert" not found
Sep 10 23:49:30.653051 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:30.653023    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls podName:13c51a1d-d12a-48da-a6ef-c88f5e897c05 nodeName:}" failed. No retries permitted until 2026-09-10 23:49:31.153001416 +0000 UTC m=+782.984028942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls") pod "maas-api-e2e-autores-74f1c1-6f8db58456-ccv42" (UID: "13c51a1d-d12a-48da-a6ef-c88f5e897c05") : secret "maas-api-e2e-autores-74f1c1-serving-cert" not found
Sep 10 23:49:30.653474 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.653457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c51a1d-d12a-48da-a6ef-c88f5e897c05-openshift-service-ca\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.655170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.655150    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6" path="/var/lib/kubelet/pods/0efdfc16-3f35-4546-bd84-6a2d7b1b1ce6/volumes"
Sep 10 23:49:30.655445 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.655430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-metrics-certs\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.661205 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.661183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lph77\" (UniqueName: \"kubernetes.io/projected/13c51a1d-d12a-48da-a6ef-c88f5e897c05-kube-api-access-lph77\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:30.771893 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.771816    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:30.867551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:30.867517    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:31.157333 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.157241    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:31.159516 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.159490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls\") pod \"maas-api-e2e-autores-74f1c1-6f8db58456-ccv42\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:31.167571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.167550    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:31.266328 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.266299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:31.273815 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.273788    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"]
Sep 10 23:49:31.278420 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.278404    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:31.281285 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.281266    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-74f1c1\""
Sep 10 23:49:31.290023 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.290003    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"]
Sep 10 23:49:31.298376 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.298351    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"]
Sep 10 23:49:31.301763 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.301745    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:31.304323 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.304305    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-74f1c1-dockercfg-g7xjb\""
Sep 10 23:49:31.317306 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.317280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"]
Sep 10 23:49:31.359628 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.359602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28a885-e231-4304-8f01-c92a3d5fad15-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb\" (UID: \"8c28a885-e231-4304-8f01-c92a3d5fad15\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:31.359759 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.359677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxpj\" (UniqueName: \"kubernetes.io/projected/8c28a885-e231-4304-8f01-c92a3d5fad15-kube-api-access-5sxpj\") pod \"payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb\" (UID: \"8c28a885-e231-4304-8f01-c92a3d5fad15\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:31.387548 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.387521    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:31.461156 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.461121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxpj\" (UniqueName: \"kubernetes.io/projected/8c28a885-e231-4304-8f01-c92a3d5fad15-kube-api-access-5sxpj\") pod \"payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb\" (UID: \"8c28a885-e231-4304-8f01-c92a3d5fad15\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:31.461297 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.461212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28a885-e231-4304-8f01-c92a3d5fad15-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb\" (UID: \"8c28a885-e231-4304-8f01-c92a3d5fad15\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:31.461297 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.461238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxx9\" (UniqueName: \"kubernetes.io/projected/1004c271-4d37-4265-ae31-9d53edcf3b8a-kube-api-access-xhxx9\") pod \"payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l\" (UID: \"1004c271-4d37-4265-ae31-9d53edcf3b8a\") " pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:31.461382 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.461301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004c271-4d37-4265-ae31-9d53edcf3b8a-plugins-config-volume\") pod \"payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l\" (UID: \"1004c271-4d37-4265-ae31-9d53edcf3b8a\") " pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:31.461895 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.461877    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28a885-e231-4304-8f01-c92a3d5fad15-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb\" (UID: \"8c28a885-e231-4304-8f01-c92a3d5fad15\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:31.469984 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.469955    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxpj\" (UniqueName: \"kubernetes.io/projected/8c28a885-e231-4304-8f01-c92a3d5fad15-kube-api-access-5sxpj\") pod \"payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb\" (UID: \"8c28a885-e231-4304-8f01-c92a3d5fad15\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:31.562513 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.562477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004c271-4d37-4265-ae31-9d53edcf3b8a-plugins-config-volume\") pod \"payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l\" (UID: \"1004c271-4d37-4265-ae31-9d53edcf3b8a\") " pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:31.562989 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.562588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxx9\" (UniqueName: \"kubernetes.io/projected/1004c271-4d37-4265-ae31-9d53edcf3b8a-kube-api-access-xhxx9\") pod \"payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l\" (UID: \"1004c271-4d37-4265-ae31-9d53edcf3b8a\") " pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:31.563199 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.563177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004c271-4d37-4265-ae31-9d53edcf3b8a-plugins-config-volume\") pod \"payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l\" (UID: \"1004c271-4d37-4265-ae31-9d53edcf3b8a\") " pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:31.570915 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.570886    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxx9\" (UniqueName: \"kubernetes.io/projected/1004c271-4d37-4265-ae31-9d53edcf3b8a-kube-api-access-xhxx9\") pod \"payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l\" (UID: \"1004c271-4d37-4265-ae31-9d53edcf3b8a\") " pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:31.588666 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.588641    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:31.614029 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.614002    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:31.667093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.667060    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"]
Sep 10 23:49:31.673233 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:49:31.673199    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c51a1d_d12a_48da_a6ef_c88f5e897c05.slice/crio-6c336428e9b6afda111dca508cf999b70c72afa5ec4ceb1c2eae34219b9d8122 WatchSource:0}: Error finding container 6c336428e9b6afda111dca508cf999b70c72afa5ec4ceb1c2eae34219b9d8122: Status 404 returned error can't find the container with id 6c336428e9b6afda111dca508cf999b70c72afa5ec4ceb1c2eae34219b9d8122
Sep 10 23:49:31.863850 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.863823    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"]
Sep 10 23:49:31.865638 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:49:31.865614    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c28a885_e231_4304_8f01_c92a3d5fad15.slice/crio-07a3cb9ebc67382ef2c3bfa07f2d4b27879a2d2245b0a34d6307081b31933a59 WatchSource:0}: Error finding container 07a3cb9ebc67382ef2c3bfa07f2d4b27879a2d2245b0a34d6307081b31933a59: Status 404 returned error can't find the container with id 07a3cb9ebc67382ef2c3bfa07f2d4b27879a2d2245b0a34d6307081b31933a59
Sep 10 23:49:31.897716 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:31.897693    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"]
Sep 10 23:49:31.902938 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:49:31.902895    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1004c271_4d37_4265_ae31_9d53edcf3b8a.slice/crio-91977c7b2d082fa63a22baa7099b390d914e4e2b239b7b6aa2fa3f6eb4918695 WatchSource:0}: Error finding container 91977c7b2d082fa63a22baa7099b390d914e4e2b239b7b6aa2fa3f6eb4918695: Status 404 returned error can't find the container with id 91977c7b2d082fa63a22baa7099b390d914e4e2b239b7b6aa2fa3f6eb4918695
Sep 10 23:49:32.202548 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.202504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l" event={"ID":"1004c271-4d37-4265-ae31-9d53edcf3b8a","Type":"ContainerStarted","Data":"2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe"}
Sep 10 23:49:32.202735 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.202576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l" event={"ID":"1004c271-4d37-4265-ae31-9d53edcf3b8a","Type":"ContainerStarted","Data":"91977c7b2d082fa63a22baa7099b390d914e4e2b239b7b6aa2fa3f6eb4918695"}
Sep 10 23:49:32.202735 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.202631    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:32.204485 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.204307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42" event={"ID":"13c51a1d-d12a-48da-a6ef-c88f5e897c05","Type":"ContainerStarted","Data":"7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377"}
Sep 10 23:49:32.204485 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.204353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42" event={"ID":"13c51a1d-d12a-48da-a6ef-c88f5e897c05","Type":"ContainerStarted","Data":"6c336428e9b6afda111dca508cf999b70c72afa5ec4ceb1c2eae34219b9d8122"}
Sep 10 23:49:32.204485 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.204388    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:32.206235 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.206164    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb" event={"ID":"8c28a885-e231-4304-8f01-c92a3d5fad15","Type":"ContainerStarted","Data":"b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83"}
Sep 10 23:49:32.206235 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.206203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb" event={"ID":"8c28a885-e231-4304-8f01-c92a3d5fad15","Type":"ContainerStarted","Data":"07a3cb9ebc67382ef2c3bfa07f2d4b27879a2d2245b0a34d6307081b31933a59"}
Sep 10 23:49:32.206415 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.206242    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:32.228537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.228494    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l" podStartSLOduration=1.228480313 podStartE2EDuration="1.228480313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:32.227534736 +0000 UTC m=+784.058562286" watchObservedRunningTime="2026-09-10 23:49:32.228480313 +0000 UTC m=+784.059507845"
Sep 10 23:49:32.249550 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.249496    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42" podStartSLOduration=1.945714511 podStartE2EDuration="2.249479649s" totalImagesPullingTime="303.765138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:32.249101309 +0000 UTC m=+784.080128867" watchObservedRunningTime="2026-09-10 23:49:32.249479649 +0000 UTC m=+784.080507184"
Sep 10 23:49:32.274983 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.274919    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb" podStartSLOduration=1.274900406 podStartE2EDuration="1.274900406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:32.272835518 +0000 UTC m=+784.103863080" watchObservedRunningTime="2026-09-10 23:49:32.274900406 +0000 UTC m=+784.105927944"
Sep 10 23:49:32.325020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.324990    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"]
Sep 10 23:49:32.328576 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.328558    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.331293 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.331265    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e027297d-serving-cert\""
Sep 10 23:49:32.346471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.346448    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"]
Sep 10 23:49:32.364943 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.364897    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:32.462572 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.462488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:32.471654 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.471626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.471789 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.471670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-openshift-service-ca\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.471789 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.471706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7wp\" (UniqueName: \"kubernetes.io/projected/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-kube-api-access-bs7wp\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.471789 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.471746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-metrics-certs\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.573187 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.573157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.573602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.573197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-openshift-service-ca\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.573602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.573221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7wp\" (UniqueName: \"kubernetes.io/projected/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-kube-api-access-bs7wp\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.573602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.573255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-metrics-certs\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.573602 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:32.573321    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-serving-cert: secret "maas-api-e2e-ait-e027297d-serving-cert" not found
Sep 10 23:49:32.573602 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:32.573391    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls podName:0734d3b7-aa87-4e3a-83b8-82ec8562ca8f nodeName:}" failed. No retries permitted until 2026-09-10 23:49:33.073376146 +0000 UTC m=+784.904403670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls") pod "maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp" (UID: "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f") : secret "maas-api-e2e-ait-e027297d-serving-cert" not found
Sep 10 23:49:32.573879 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.573855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-openshift-service-ca\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.575774 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.575757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-metrics-certs\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.583942 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.583913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7wp\" (UniqueName: \"kubernetes.io/projected/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-kube-api-access-bs7wp\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:32.854174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.854082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:32.954663 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:32.954624    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:33.078242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:33.078212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:33.078418 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:33.078380    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-serving-cert: secret "maas-api-e2e-ait-e027297d-serving-cert" not found
Sep 10 23:49:33.078491 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:49:33.078454    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls podName:0734d3b7-aa87-4e3a-83b8-82ec8562ca8f nodeName:}" failed. No retries permitted until 2026-09-10 23:49:34.078434735 +0000 UTC m=+785.909462263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls") pod "maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp" (UID: "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f") : secret "maas-api-e2e-ait-e027297d-serving-cert" not found
Sep 10 23:49:33.157535 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:33.157455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:33.181415 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:33.181386    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:49:34.087355 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:34.087322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:34.089603 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:34.089582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls\") pod \"maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:34.140572 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:34.140547    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:34.414210 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:34.414173    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"]
Sep 10 23:49:34.415634 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:49:34.415605    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0734d3b7_aa87_4e3a_83b8_82ec8562ca8f.slice/crio-90deb6271086deed9207b57bee1bb8008c43b8dbfff48bb1f9db0bd579a82cf6 WatchSource:0}: Error finding container 90deb6271086deed9207b57bee1bb8008c43b8dbfff48bb1f9db0bd579a82cf6: Status 404 returned error can't find the container with id 90deb6271086deed9207b57bee1bb8008c43b8dbfff48bb1f9db0bd579a82cf6
Sep 10 23:49:35.219892 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:35.219855    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp" event={"ID":"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f","Type":"ContainerStarted","Data":"b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010"}
Sep 10 23:49:35.220364 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:35.219980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp" event={"ID":"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f","Type":"ContainerStarted","Data":"90deb6271086deed9207b57bee1bb8008c43b8dbfff48bb1f9db0bd579a82cf6"}
Sep 10 23:49:35.220364 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:35.220020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:35.239130 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:35.239087    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp" podStartSLOduration=2.901879851 podStartE2EDuration="3.239072177s" totalImagesPullingTime="337.192326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:35.237471847 +0000 UTC m=+787.068499399" watchObservedRunningTime="2026-09-10 23:49:35.239072177 +0000 UTC m=+787.070099709"
Sep 10 23:49:37.172297 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:37.172270    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:49:38.149000 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:38.148965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:38.215673 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:38.215632    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:49:38.279653 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:38.279607    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:38.417100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:38.417020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:38.639291 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:38.639259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:40.691498 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:40.691464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:40.788434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:40.788403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:40.895191 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:40.895163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:41.190875 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:41.190844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:41.229278 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:41.229251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:49:42.835579 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:42.835551    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:43.212837 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:43.212812    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:49:43.213019 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:43.212938    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:49:46.487182 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:46.487154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:46.793811 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:46.793740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:46.905373 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:46.905342    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:46.998427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:46.998400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:47.126705 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:47.126636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:48.094050 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:48.094016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:48.444140 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:48.444110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:48.806434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:48.806351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:49.150343 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:49.150269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:51.572724 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:51.572647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:52.476600 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:52.476558    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:52.562922 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:52.562885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:53.655079 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:53.655038    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:53.913150 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:53.913083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:54.199073 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:54.199035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:54.382861 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:54.382828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:54.961081 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:54.961051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:55.057859 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:55.057829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:55.154036 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:55.154006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:55.465141 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:55.465107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:55.570890 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:55.570854    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:56.040278 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:56.040246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:56.878441 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:56.878408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:49:57.644768 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:49:57.644737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:05.753098 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.753064    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"]
Sep 10 23:50:05.757999 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.757978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.761137 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.761116    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-e6d58266-kserve-self-signed-certs\""
Sep 10 23:50:05.776091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.771830    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"]
Sep 10 23:50:05.846155 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.846118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-tmp-dir\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.846155 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.846155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-model-cache\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.846351 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.846177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6744-7e3e-4077-81f8-1c9a82d55527-tls-certs\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.846351 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.846200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54nl2\" (UniqueName: \"kubernetes.io/projected/cecd6744-7e3e-4077-81f8-1c9a82d55527-kube-api-access-54nl2\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.846351 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.846222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-home\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.846351 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.846272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-dshm\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.946740 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.946705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-tmp-dir\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.946740 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.946740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-model-cache\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.947025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.946772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6744-7e3e-4077-81f8-1c9a82d55527-tls-certs\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.947025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.946799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54nl2\" (UniqueName: \"kubernetes.io/projected/cecd6744-7e3e-4077-81f8-1c9a82d55527-kube-api-access-54nl2\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.947157 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.947092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-home\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.947157 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.947137    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-tmp-dir\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.947264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.947175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-dshm\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.947264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.947183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-model-cache\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.947425 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.947405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-home\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.949190 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.949168    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-dshm\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.949386 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.949365    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6744-7e3e-4077-81f8-1c9a82d55527-tls-certs\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:05.956531 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:05.956507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54nl2\" (UniqueName: \"kubernetes.io/projected/cecd6744-7e3e-4077-81f8-1c9a82d55527-kube-api-access-54nl2\") pod \"e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") " pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:06.067390 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:06.067293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:06.264516 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:06.264477    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:06.343566 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:06.343535    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"]
Sep 10 23:50:06.344663 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:50:06.344638    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcecd6744_7e3e_4077_81f8_1c9a82d55527.slice/crio-eda09c97189737d551f7ce53cc8792f705be04bf788ce510929afbb32e4a8ece WatchSource:0}: Error finding container eda09c97189737d551f7ce53cc8792f705be04bf788ce510929afbb32e4a8ece: Status 404 returned error can't find the container with id eda09c97189737d551f7ce53cc8792f705be04bf788ce510929afbb32e4a8ece
Sep 10 23:50:06.371520 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:06.371496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn" event={"ID":"cecd6744-7e3e-4077-81f8-1c9a82d55527","Type":"ContainerStarted","Data":"eda09c97189737d551f7ce53cc8792f705be04bf788ce510929afbb32e4a8ece"}
Sep 10 23:50:07.378394 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:07.378355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn" event={"ID":"cecd6744-7e3e-4077-81f8-1c9a82d55527","Type":"ContainerStarted","Data":"0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1"}
Sep 10 23:50:07.396850 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:07.396805    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn" podStartSLOduration=2.396790659 podStartE2EDuration="2.396790659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:07.395905252 +0000 UTC m=+819.226932787" watchObservedRunningTime="2026-09-10 23:50:07.396790659 +0000 UTC m=+819.227818221"
Sep 10 23:50:12.665073 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:12.665042    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"]
Sep 10 23:50:12.665634 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:12.665291    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp" podUID="0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" containerName="maas-api" containerID="cri-o://b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010" gracePeriod=30
Sep 10 23:50:12.700485 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:12.700449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:12.781181 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:12.781147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:12.847871 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:12.847835    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:12.909426 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:12.909403    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:50:12.934206 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:12.934175    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:13.011055 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.011031    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-metrics-certs\") pod \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") "
Sep 10 23:50:13.011205 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.011096    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs7wp\" (UniqueName: \"kubernetes.io/projected/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-kube-api-access-bs7wp\") pod \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") "
Sep 10 23:50:13.011205 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.011160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls\") pod \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") "
Sep 10 23:50:13.011205 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.011194    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-openshift-service-ca\") pod \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\" (UID: \"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f\") "
Sep 10 23:50:13.011645 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.011611    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" (UID: "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:50:13.013345 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.013316    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-kube-api-access-bs7wp" (OuterVolumeSpecName: "kube-api-access-bs7wp") pod "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" (UID: "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f"). InnerVolumeSpecName "kube-api-access-bs7wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:50:13.013345 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.013329    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" (UID: "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:50:13.013472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.013430    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" (UID: "0734d3b7-aa87-4e3a-83b8-82ec8562ca8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:50:13.025196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.025150    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:13.108868 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.108819    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:13.112775 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.112750    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:50:13.112775 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.112772    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:50:13.112979 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.112782    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:50:13.112979 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.112792    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs7wp\" (UniqueName: \"kubernetes.io/projected/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f-kube-api-access-bs7wp\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:50:13.408894 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.408810    2562 generic.go:356] "Generic (PLEG): container finished" podID="0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" containerID="b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010" exitCode=0
Sep 10 23:50:13.408894 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.408860    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"
Sep 10 23:50:13.409149 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.408894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp" event={"ID":"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f","Type":"ContainerDied","Data":"b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010"}
Sep 10 23:50:13.409149 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.408957    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp" event={"ID":"0734d3b7-aa87-4e3a-83b8-82ec8562ca8f","Type":"ContainerDied","Data":"90deb6271086deed9207b57bee1bb8008c43b8dbfff48bb1f9db0bd579a82cf6"}
Sep 10 23:50:13.409149 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.408980    2562 scope.go:117] "RemoveContainer" containerID="b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010"
Sep 10 23:50:13.418427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.418409    2562 scope.go:117] "RemoveContainer" containerID="b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010"
Sep 10 23:50:13.418664 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:50:13.418649    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010\": container with ID starting with b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010 not found: ID does not exist" containerID="b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010"
Sep 10 23:50:13.418710 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.418670    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010"} err="failed to get container status \"b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010\": rpc error: code = NotFound desc = could not find container \"b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010\": container with ID starting with b7b9db5cd233441364b955fe4c17b3a32c291451df4302eb568e4067cb5d2010 not found: ID does not exist"
Sep 10 23:50:13.429775 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.429752    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"]
Sep 10 23:50:13.435807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.435783    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e027297d-cd4dcdf99-f4jzp"]
Sep 10 23:50:13.534283 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.534248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:13.632388 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.632364    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:13.731751 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:13.731724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:14.189095 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:14.189064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:14.371658 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:14.371629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:14.655792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:14.655707    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" path="/var/lib/kubelet/pods/0734d3b7-aa87-4e3a-83b8-82ec8562ca8f/volumes"
Sep 10 23:50:16.067861 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:16.067828    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:16.068367 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:16.067887    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:16.080457 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:16.080435    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:16.135230 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:16.135204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:16.434511 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:16.434485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:50:16.685093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:16.685022    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:17.410664 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.410629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:17.521490 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.521454    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:17.949505 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.949475    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"]
Sep 10 23:50:17.949891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.949877    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" containerName="maas-api"
Sep 10 23:50:17.949993 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.949892    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" containerName="maas-api"
Sep 10 23:50:17.949993 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.949989    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0734d3b7-aa87-4e3a-83b8-82ec8562ca8f" containerName="maas-api"
Sep 10 23:50:17.954323 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.954303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:17.958262 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.958239    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6a8995da-serving-cert\""
Sep 10 23:50:17.971352 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:17.971323    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"]
Sep 10 23:50:18.002247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.002214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:18.052962 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.052917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.053077 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.052976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-metrics-certs\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.053127 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.053084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23753cd1-24d8-407f-b74c-8f14647d5647-openshift-service-ca\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.053127 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.053117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlx9s\" (UniqueName: \"kubernetes.io/projected/23753cd1-24d8-407f-b74c-8f14647d5647-kube-api-access-xlx9s\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.108513 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.108490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:18.153801 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.153772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23753cd1-24d8-407f-b74c-8f14647d5647-openshift-service-ca\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.153910 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.153811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlx9s\" (UniqueName: \"kubernetes.io/projected/23753cd1-24d8-407f-b74c-8f14647d5647-kube-api-access-xlx9s\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.153910 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.153867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.154057 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.153913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-metrics-certs\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.154057 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:50:18.154012    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-serving-cert: secret "maas-api-e2e-migrate-6a8995da-serving-cert" not found
Sep 10 23:50:18.154140 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:50:18.154087    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls podName:23753cd1-24d8-407f-b74c-8f14647d5647 nodeName:}" failed. No retries permitted until 2026-09-10 23:50:18.6540665 +0000 UTC m=+830.485094034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls") pod "maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl" (UID: "23753cd1-24d8-407f-b74c-8f14647d5647") : secret "maas-api-e2e-migrate-6a8995da-serving-cert" not found
Sep 10 23:50:18.154401 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.154383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23753cd1-24d8-407f-b74c-8f14647d5647-openshift-service-ca\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.156461 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.156442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-metrics-certs\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.162799 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.162776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlx9s\" (UniqueName: \"kubernetes.io/projected/23753cd1-24d8-407f-b74c-8f14647d5647-kube-api-access-xlx9s\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.531960 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.531921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:18.637156 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.637102    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:18.658675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.658649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.661061 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.661041    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls\") pod \"maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:18.864146 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:18.864068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:19.138529 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:19.138505    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"]
Sep 10 23:50:19.140860 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:50:19.140823    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23753cd1_24d8_407f_b74c_8f14647d5647.slice/crio-b96ca38faf088b6c655126dbd27b393e1d6f71c94fe87c0f6c46aa22c8faf003 WatchSource:0}: Error finding container b96ca38faf088b6c655126dbd27b393e1d6f71c94fe87c0f6c46aa22c8faf003: Status 404 returned error can't find the container with id b96ca38faf088b6c655126dbd27b393e1d6f71c94fe87c0f6c46aa22c8faf003
Sep 10 23:50:19.155852 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:19.155826    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:19.437980 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:19.437950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl" event={"ID":"23753cd1-24d8-407f-b74c-8f14647d5647","Type":"ContainerStarted","Data":"b96ca38faf088b6c655126dbd27b393e1d6f71c94fe87c0f6c46aa22c8faf003"}
Sep 10 23:50:20.080105 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:20.080073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:20.447275 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:20.447246    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl" event={"ID":"23753cd1-24d8-407f-b74c-8f14647d5647","Type":"ContainerStarted","Data":"a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212"}
Sep 10 23:50:20.447453 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:20.447355    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:20.466458 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:20.466414    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl" podStartSLOduration=3.118243876 podStartE2EDuration="3.466398783s" totalImagesPullingTime="348.154907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:20.465331445 +0000 UTC m=+832.296359003" watchObservedRunningTime="2026-09-10 23:50:20.466398783 +0000 UTC m=+832.297426327"
Sep 10 23:50:22.453006 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:22.452975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:26.458443 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:26.458413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:39.740155 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:39.740121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:48.471914 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.471879    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"]
Sep 10 23:50:48.472434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.472149    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl" podUID="23753cd1-24d8-407f-b74c-8f14647d5647" containerName="maas-api" containerID="cri-o://a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212" gracePeriod=30
Sep 10 23:50:48.562993 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.562957    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:48.736148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.736123    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:48.742288 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.742266    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:48.819858 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.819829    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls\") pod \"23753cd1-24d8-407f-b74c-8f14647d5647\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") "
Sep 10 23:50:48.820031 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.819879    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-metrics-certs\") pod \"23753cd1-24d8-407f-b74c-8f14647d5647\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") "
Sep 10 23:50:48.820031 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.819946    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlx9s\" (UniqueName: \"kubernetes.io/projected/23753cd1-24d8-407f-b74c-8f14647d5647-kube-api-access-xlx9s\") pod \"23753cd1-24d8-407f-b74c-8f14647d5647\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") "
Sep 10 23:50:48.820031 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.819975    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23753cd1-24d8-407f-b74c-8f14647d5647-openshift-service-ca\") pod \"23753cd1-24d8-407f-b74c-8f14647d5647\" (UID: \"23753cd1-24d8-407f-b74c-8f14647d5647\") "
Sep 10 23:50:48.820400 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.820367    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23753cd1-24d8-407f-b74c-8f14647d5647-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23753cd1-24d8-407f-b74c-8f14647d5647" (UID: "23753cd1-24d8-407f-b74c-8f14647d5647"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:50:48.822100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.822064    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "23753cd1-24d8-407f-b74c-8f14647d5647" (UID: "23753cd1-24d8-407f-b74c-8f14647d5647"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:50:48.822208 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.822107    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23753cd1-24d8-407f-b74c-8f14647d5647" (UID: "23753cd1-24d8-407f-b74c-8f14647d5647"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:50:48.822274 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.822244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23753cd1-24d8-407f-b74c-8f14647d5647-kube-api-access-xlx9s" (OuterVolumeSpecName: "kube-api-access-xlx9s") pod "23753cd1-24d8-407f-b74c-8f14647d5647" (UID: "23753cd1-24d8-407f-b74c-8f14647d5647"). InnerVolumeSpecName "kube-api-access-xlx9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:50:48.826519 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.826497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:48.921616 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.921589    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:50:48.921616 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.921611    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23753cd1-24d8-407f-b74c-8f14647d5647-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:50:48.921616 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.921620    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlx9s\" (UniqueName: \"kubernetes.io/projected/23753cd1-24d8-407f-b74c-8f14647d5647-kube-api-access-xlx9s\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:50:48.921803 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.921629    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23753cd1-24d8-407f-b74c-8f14647d5647-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:50:48.932960 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:48.932913    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:49.018286 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.018221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:49.107332 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.107298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:49.520138 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.520102    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:49.591137 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.591100    2562 generic.go:356] "Generic (PLEG): container finished" podID="23753cd1-24d8-407f-b74c-8f14647d5647" containerID="a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212" exitCode=0
Sep 10 23:50:49.591281 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.591148    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"
Sep 10 23:50:49.591281 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.591150    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl" event={"ID":"23753cd1-24d8-407f-b74c-8f14647d5647","Type":"ContainerDied","Data":"a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212"}
Sep 10 23:50:49.591358 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.591278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl" event={"ID":"23753cd1-24d8-407f-b74c-8f14647d5647","Type":"ContainerDied","Data":"b96ca38faf088b6c655126dbd27b393e1d6f71c94fe87c0f6c46aa22c8faf003"}
Sep 10 23:50:49.591358 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.591314    2562 scope.go:117] "RemoveContainer" containerID="a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212"
Sep 10 23:50:49.600707 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.600657    2562 scope.go:117] "RemoveContainer" containerID="a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212"
Sep 10 23:50:49.600947 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:50:49.600914    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212\": container with ID starting with a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212 not found: ID does not exist" containerID="a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212"
Sep 10 23:50:49.601035 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.600955    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212"} err="failed to get container status \"a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212\": rpc error: code = NotFound desc = could not find container \"a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212\": container with ID starting with a90e6b82db8a13f2855bf03b20bf59518363d974538d32ffe8da8fafd8be7212 not found: ID does not exist"
Sep 10 23:50:49.612494 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.612473    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"]
Sep 10 23:50:49.616054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.616030    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6a8995da-6c75484cdf-g9lpl"]
Sep 10 23:50:49.813306 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:49.813237    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:50.038226 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:50.038195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:50.655126 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:50.655093    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23753cd1-24d8-407f-b74c-8f14647d5647" path="/var/lib/kubelet/pods/23753cd1-24d8-407f-b74c-8f14647d5647/volumes"
Sep 10 23:50:52.450478 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:52.450446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:52.546173 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:52.546141    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:53.709503 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.709468    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"]
Sep 10 23:50:53.710033 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.709877    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23753cd1-24d8-407f-b74c-8f14647d5647" containerName="maas-api"
Sep 10 23:50:53.710033 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.709890    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="23753cd1-24d8-407f-b74c-8f14647d5647" containerName="maas-api"
Sep 10 23:50:53.710033 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.709994    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="23753cd1-24d8-407f-b74c-8f14647d5647" containerName="maas-api"
Sep 10 23:50:53.714644 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.714619    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.717059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.717039    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5706c553-serving-cert\""
Sep 10 23:50:53.724236 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.724207    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"]
Sep 10 23:50:53.744344 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.744322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:53.764009 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.763978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xr4\" (UniqueName: \"kubernetes.io/projected/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-kube-api-access-d8xr4\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.764105 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.764060    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-openshift-service-ca\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.764147 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.764113    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.764147 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.764140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-metrics-certs\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.812525 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.812498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:53.865007 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.864977    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.865110 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.865011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-metrics-certs\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.865110 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.865050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xr4\" (UniqueName: \"kubernetes.io/projected/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-kube-api-access-d8xr4\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.865206 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.865109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-openshift-service-ca\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.865206 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:50:53.865143    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-serving-cert: secret "maas-api-e2e-ait-5706c553-serving-cert" not found
Sep 10 23:50:53.865206 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:50:53.865203    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls podName:df6b9ea8-7872-4d7d-9875-61a1a77ccdfd nodeName:}" failed. No retries permitted until 2026-09-10 23:50:54.365184822 +0000 UTC m=+866.196212348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls") pod "maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8" (UID: "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd") : secret "maas-api-e2e-ait-5706c553-serving-cert" not found
Sep 10 23:50:53.865706 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.865687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-openshift-service-ca\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.867305 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.867285    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-metrics-certs\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:53.874227 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:53.874203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xr4\" (UniqueName: \"kubernetes.io/projected/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-kube-api-access-d8xr4\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:54.086476 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:54.086399    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:54.321411 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:54.321380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:54.369788 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:54.369717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:54.372051 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:54.372033    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls\") pod \"maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:54.425521 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:54.425497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:50:54.628804 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:54.628730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:54.901942 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:50:54.901901    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6b9ea8_7872_4d7d_9875_61a1a77ccdfd.slice/crio-f98b707ffda54188605500a29cdbfd2450ed893b85ec2cdbd942794676916fd6 WatchSource:0}: Error finding container f98b707ffda54188605500a29cdbfd2450ed893b85ec2cdbd942794676916fd6: Status 404 returned error can't find the container with id f98b707ffda54188605500a29cdbfd2450ed893b85ec2cdbd942794676916fd6
Sep 10 23:50:54.904185 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:54.904153    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"]
Sep 10 23:50:55.622215 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:55.622177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8" event={"ID":"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd","Type":"ContainerStarted","Data":"e485cd300232740ae8148d553d95a99fe7fbfe4963fa8d972225619940bba5d1"}
Sep 10 23:50:55.622378 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:55.622245    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:50:55.622378 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:55.622268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8" event={"ID":"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd","Type":"ContainerStarted","Data":"f98b707ffda54188605500a29cdbfd2450ed893b85ec2cdbd942794676916fd6"}
Sep 10 23:50:55.653009 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:50:55.652904    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8" podStartSLOduration=2.176320848 podStartE2EDuration="2.652888231s" totalImagesPullingTime="476.567383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:55.65282774 +0000 UTC m=+867.483855292" watchObservedRunningTime="2026-09-10 23:50:55.652888231 +0000 UTC m=+867.483915764"
Sep 10 23:51:01.633019 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:01.632985    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:51:12.244199 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:12.244163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:13.153486 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:13.153450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:14.713481 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:14.713449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:22.497406 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:22.497324    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:23.096470 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.096439    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"]
Sep 10 23:51:23.096722 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.096683    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42" podUID="13c51a1d-d12a-48da-a6ef-c88f5e897c05" containerName="maas-api" containerID="cri-o://7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377" gracePeriod=30
Sep 10 23:51:23.123471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.123437    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:23.150632 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.150601    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"]
Sep 10 23:51:23.150842 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.150815    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l" podUID="1004c271-4d37-4265-ae31-9d53edcf3b8a" containerName="payload-processing" containerID="cri-o://2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe" gracePeriod=30
Sep 10 23:51:23.159891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.159866    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"]
Sep 10 23:51:23.160122 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.160093    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb" podUID="8c28a885-e231-4304-8f01-c92a3d5fad15" containerName="payload-pre-processing" containerID="cri-o://b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83" gracePeriod=30
Sep 10 23:51:23.210439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.210406    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l" podUID="1004c271-4d37-4265-ae31-9d53edcf3b8a" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.47:9004: connect: connection refused"
Sep 10 23:51:23.210439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.210420    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb" podUID="8c28a885-e231-4304-8f01-c92a3d5fad15" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.46:9004: connect: connection refused"
Sep 10 23:51:23.210603 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.210421    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42" podUID="13c51a1d-d12a-48da-a6ef-c88f5e897c05" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.45:8443/health\": dial tcp 10.132.0.45:8443: connect: connection refused"
Sep 10 23:51:23.210975 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.210953    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:23.340995 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.340960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:23.417426 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.417403    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:51:23.443471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.443422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:23.463831 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.463810    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:51:23.471214 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.471192    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:51:23.525729 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.525705    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c51a1d-d12a-48da-a6ef-c88f5e897c05-openshift-service-ca\") pod \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") "
Sep 10 23:51:23.526059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.525788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-metrics-certs\") pod \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") "
Sep 10 23:51:23.526059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.525883    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lph77\" (UniqueName: \"kubernetes.io/projected/13c51a1d-d12a-48da-a6ef-c88f5e897c05-kube-api-access-lph77\") pod \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") "
Sep 10 23:51:23.526059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.525948    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls\") pod \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\" (UID: \"13c51a1d-d12a-48da-a6ef-c88f5e897c05\") "
Sep 10 23:51:23.526174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.526074    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c51a1d-d12a-48da-a6ef-c88f5e897c05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13c51a1d-d12a-48da-a6ef-c88f5e897c05" (UID: "13c51a1d-d12a-48da-a6ef-c88f5e897c05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:23.526318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.526297    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c51a1d-d12a-48da-a6ef-c88f5e897c05-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:23.527859 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.527835    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "13c51a1d-d12a-48da-a6ef-c88f5e897c05" (UID: "13c51a1d-d12a-48da-a6ef-c88f5e897c05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:51:23.527953 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.527876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13c51a1d-d12a-48da-a6ef-c88f5e897c05" (UID: "13c51a1d-d12a-48da-a6ef-c88f5e897c05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:51:23.527953 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.527901    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c51a1d-d12a-48da-a6ef-c88f5e897c05-kube-api-access-lph77" (OuterVolumeSpecName: "kube-api-access-lph77") pod "13c51a1d-d12a-48da-a6ef-c88f5e897c05" (UID: "13c51a1d-d12a-48da-a6ef-c88f5e897c05"). InnerVolumeSpecName "kube-api-access-lph77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:23.627336 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627310    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004c271-4d37-4265-ae31-9d53edcf3b8a-plugins-config-volume\") pod \"1004c271-4d37-4265-ae31-9d53edcf3b8a\" (UID: \"1004c271-4d37-4265-ae31-9d53edcf3b8a\") "
Sep 10 23:51:23.627336 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627338    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhxx9\" (UniqueName: \"kubernetes.io/projected/1004c271-4d37-4265-ae31-9d53edcf3b8a-kube-api-access-xhxx9\") pod \"1004c271-4d37-4265-ae31-9d53edcf3b8a\" (UID: \"1004c271-4d37-4265-ae31-9d53edcf3b8a\") "
Sep 10 23:51:23.627491 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627393    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sxpj\" (UniqueName: \"kubernetes.io/projected/8c28a885-e231-4304-8f01-c92a3d5fad15-kube-api-access-5sxpj\") pod \"8c28a885-e231-4304-8f01-c92a3d5fad15\" (UID: \"8c28a885-e231-4304-8f01-c92a3d5fad15\") "
Sep 10 23:51:23.627528 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627492    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28a885-e231-4304-8f01-c92a3d5fad15-plugins-config-volume\") pod \"8c28a885-e231-4304-8f01-c92a3d5fad15\" (UID: \"8c28a885-e231-4304-8f01-c92a3d5fad15\") "
Sep 10 23:51:23.627726 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627709    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:23.627783 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627727    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lph77\" (UniqueName: \"kubernetes.io/projected/13c51a1d-d12a-48da-a6ef-c88f5e897c05-kube-api-access-lph77\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:23.627783 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627738    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c51a1d-d12a-48da-a6ef-c88f5e897c05-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:23.627890 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1004c271-4d37-4265-ae31-9d53edcf3b8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1004c271-4d37-4265-ae31-9d53edcf3b8a" (UID: "1004c271-4d37-4265-ae31-9d53edcf3b8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:23.627890 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.627819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c28a885-e231-4304-8f01-c92a3d5fad15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c28a885-e231-4304-8f01-c92a3d5fad15" (UID: "8c28a885-e231-4304-8f01-c92a3d5fad15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:23.629492 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.629469    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1004c271-4d37-4265-ae31-9d53edcf3b8a-kube-api-access-xhxx9" (OuterVolumeSpecName: "kube-api-access-xhxx9") pod "1004c271-4d37-4265-ae31-9d53edcf3b8a" (UID: "1004c271-4d37-4265-ae31-9d53edcf3b8a"). InnerVolumeSpecName "kube-api-access-xhxx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:23.629492 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.629471    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c28a885-e231-4304-8f01-c92a3d5fad15-kube-api-access-5sxpj" (OuterVolumeSpecName: "kube-api-access-5sxpj") pod "8c28a885-e231-4304-8f01-c92a3d5fad15" (UID: "8c28a885-e231-4304-8f01-c92a3d5fad15"). InnerVolumeSpecName "kube-api-access-5sxpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:23.728847 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.728826    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c28a885-e231-4304-8f01-c92a3d5fad15-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:23.728847 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.728847    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004c271-4d37-4265-ae31-9d53edcf3b8a-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:23.728999 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.728856    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxx9\" (UniqueName: \"kubernetes.io/projected/1004c271-4d37-4265-ae31-9d53edcf3b8a-kube-api-access-xhxx9\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:23.728999 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.728866    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxpj\" (UniqueName: \"kubernetes.io/projected/8c28a885-e231-4304-8f01-c92a3d5fad15-kube-api-access-5sxpj\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:23.745067 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.745046    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:23.753487 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.753464    2562 generic.go:356] "Generic (PLEG): container finished" podID="1004c271-4d37-4265-ae31-9d53edcf3b8a" containerID="2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe" exitCode=0
Sep 10 23:51:23.753570 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.753505    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"
Sep 10 23:51:23.753570 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.753546    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l" event={"ID":"1004c271-4d37-4265-ae31-9d53edcf3b8a","Type":"ContainerDied","Data":"2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe"}
Sep 10 23:51:23.753671 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.753596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l" event={"ID":"1004c271-4d37-4265-ae31-9d53edcf3b8a","Type":"ContainerDied","Data":"91977c7b2d082fa63a22baa7099b390d914e4e2b239b7b6aa2fa3f6eb4918695"}
Sep 10 23:51:23.753671 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.753627    2562 scope.go:117] "RemoveContainer" containerID="2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe"
Sep 10 23:51:23.754917 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.754896    2562 generic.go:356] "Generic (PLEG): container finished" podID="13c51a1d-d12a-48da-a6ef-c88f5e897c05" containerID="7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377" exitCode=0
Sep 10 23:51:23.755012 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.754962    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"
Sep 10 23:51:23.755098 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.754958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42" event={"ID":"13c51a1d-d12a-48da-a6ef-c88f5e897c05","Type":"ContainerDied","Data":"7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377"}
Sep 10 23:51:23.755155 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.755130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42" event={"ID":"13c51a1d-d12a-48da-a6ef-c88f5e897c05","Type":"ContainerDied","Data":"6c336428e9b6afda111dca508cf999b70c72afa5ec4ceb1c2eae34219b9d8122"}
Sep 10 23:51:23.756527 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.756504    2562 generic.go:356] "Generic (PLEG): container finished" podID="8c28a885-e231-4304-8f01-c92a3d5fad15" containerID="b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83" exitCode=0
Sep 10 23:51:23.756627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.756543    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb" event={"ID":"8c28a885-e231-4304-8f01-c92a3d5fad15","Type":"ContainerDied","Data":"b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83"}
Sep 10 23:51:23.756627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.756560    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"
Sep 10 23:51:23.756627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.756592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb" event={"ID":"8c28a885-e231-4304-8f01-c92a3d5fad15","Type":"ContainerDied","Data":"07a3cb9ebc67382ef2c3bfa07f2d4b27879a2d2245b0a34d6307081b31933a59"}
Sep 10 23:51:23.763054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.763008    2562 scope.go:117] "RemoveContainer" containerID="2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe"
Sep 10 23:51:23.763360 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:51:23.763340    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe\": container with ID starting with 2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe not found: ID does not exist" containerID="2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe"
Sep 10 23:51:23.763449 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.763371    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe"} err="failed to get container status \"2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe\": rpc error: code = NotFound desc = could not find container \"2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe\": container with ID starting with 2e7542cb5a7321405ce412ad3020d06b59b5fd1bf20aaca79945172b257fafbe not found: ID does not exist"
Sep 10 23:51:23.763449 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.763393    2562 scope.go:117] "RemoveContainer" containerID="7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377"
Sep 10 23:51:23.773126 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.773108    2562 scope.go:117] "RemoveContainer" containerID="7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377"
Sep 10 23:51:23.773380 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:51:23.773359    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377\": container with ID starting with 7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377 not found: ID does not exist" containerID="7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377"
Sep 10 23:51:23.773422 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.773387    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377"} err="failed to get container status \"7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377\": rpc error: code = NotFound desc = could not find container \"7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377\": container with ID starting with 7825c2000107eb1d6585c972eddca4641eecb39e41ca32e2aea851bb243b9377 not found: ID does not exist"
Sep 10 23:51:23.773422 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.773403    2562 scope.go:117] "RemoveContainer" containerID="b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83"
Sep 10 23:51:23.780702 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.780685    2562 scope.go:117] "RemoveContainer" containerID="b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83"
Sep 10 23:51:23.780972 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:51:23.780949    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83\": container with ID starting with b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83 not found: ID does not exist" containerID="b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83"
Sep 10 23:51:23.781025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.780981    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83"} err="failed to get container status \"b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83\": rpc error: code = NotFound desc = could not find container \"b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83\": container with ID starting with b14365f2da9ce22e76ee604a9696903bcdd9f621d68d10800e208e65220f0f83 not found: ID does not exist"
Sep 10 23:51:23.787784 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.787765    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"]
Sep 10 23:51:23.798780 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.798753    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-74f1c1-6f8db58456-ccv42"]
Sep 10 23:51:23.811750 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.811732    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"]
Sep 10 23:51:23.825727 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.825707    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-74f1c1-5d4c9b7765-49h6l"]
Sep 10 23:51:23.836868 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.836846    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"]
Sep 10 23:51:23.844527 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.844504    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-74f1c1-867c859dc4-89rgb"]
Sep 10 23:51:23.955230 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:23.955199    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:24.528555 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.528521    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"]
Sep 10 23:51:24.528975 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.528756    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8" podUID="df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" containerName="maas-api" containerID="cri-o://e485cd300232740ae8148d553d95a99fe7fbfe4963fa8d972225619940bba5d1" gracePeriod=30
Sep 10 23:51:24.553495 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.553471    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:24.623025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.622981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:24.655756 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.655723    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1004c271-4d37-4265-ae31-9d53edcf3b8a" path="/var/lib/kubelet/pods/1004c271-4d37-4265-ae31-9d53edcf3b8a/volumes"
Sep 10 23:51:24.656200 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.656178    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c51a1d-d12a-48da-a6ef-c88f5e897c05" path="/var/lib/kubelet/pods/13c51a1d-d12a-48da-a6ef-c88f5e897c05/volumes"
Sep 10 23:51:24.656585 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.656566    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c28a885-e231-4304-8f01-c92a3d5fad15" path="/var/lib/kubelet/pods/8c28a885-e231-4304-8f01-c92a3d5fad15/volumes"
Sep 10 23:51:24.686530 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.686502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:24.767795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.767768    2562 generic.go:356] "Generic (PLEG): container finished" podID="df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" containerID="e485cd300232740ae8148d553d95a99fe7fbfe4963fa8d972225619940bba5d1" exitCode=0
Sep 10 23:51:24.767902 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.767802    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8" event={"ID":"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd","Type":"ContainerDied","Data":"e485cd300232740ae8148d553d95a99fe7fbfe4963fa8d972225619940bba5d1"}
Sep 10 23:51:24.769267 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.769246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:24.782536 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.782485    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:51:24.828314 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.828285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:24.838200 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.838182    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls\") pod \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") "
Sep 10 23:51:24.838290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.838234    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-openshift-service-ca\") pod \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") "
Sep 10 23:51:24.838290 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.838259    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-metrics-certs\") pod \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") "
Sep 10 23:51:24.838439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.838418    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8xr4\" (UniqueName: \"kubernetes.io/projected/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-kube-api-access-d8xr4\") pod \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\" (UID: \"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd\") "
Sep 10 23:51:24.838643 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.838613    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" (UID: "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:24.838753 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.838696    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:24.840495 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.840474    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-kube-api-access-d8xr4" (OuterVolumeSpecName: "kube-api-access-d8xr4") pod "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" (UID: "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd"). InnerVolumeSpecName "kube-api-access-d8xr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:24.840794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.840775    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" (UID: "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:51:24.840854 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.840801    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" (UID: "df6b9ea8-7872-4d7d-9875-61a1a77ccdfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:51:24.939285 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.939262    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:24.939285 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.939286    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:24.939430 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:24.939299    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8xr4\" (UniqueName: \"kubernetes.io/projected/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd-kube-api-access-d8xr4\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:51:25.774776 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:25.774742    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"
Sep 10 23:51:25.775197 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:25.774775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8" event={"ID":"df6b9ea8-7872-4d7d-9875-61a1a77ccdfd","Type":"ContainerDied","Data":"f98b707ffda54188605500a29cdbfd2450ed893b85ec2cdbd942794676916fd6"}
Sep 10 23:51:25.775197 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:25.774839    2562 scope.go:117] "RemoveContainer" containerID="e485cd300232740ae8148d553d95a99fe7fbfe4963fa8d972225619940bba5d1"
Sep 10 23:51:25.796336 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:25.796310    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"]
Sep 10 23:51:25.802228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:25.802203    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5706c553-65d9cbdbbf-q5sj8"]
Sep 10 23:51:25.887199 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:25.887170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:26.282011 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:26.281980    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:26.591418 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:26.591313    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:26.655886 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:26.655845    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" path="/var/lib/kubelet/pods/df6b9ea8-7872-4d7d-9875-61a1a77ccdfd/volumes"
Sep 10 23:51:26.680346 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:26.680315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:28.019922 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:28.019887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:28.598943 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:28.598907    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:51:28.601534 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:28.601510    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:51:28.842207 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:28.842172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:29.574225 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:29.574195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:29.670690 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:29.670655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:30.680542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.680507    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"]
Sep 10 23:51:30.681134 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681113    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1004c271-4d37-4265-ae31-9d53edcf3b8a" containerName="payload-processing"
Sep 10 23:51:30.681240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681137    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1004c271-4d37-4265-ae31-9d53edcf3b8a" containerName="payload-processing"
Sep 10 23:51:30.681240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681157    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" containerName="maas-api"
Sep 10 23:51:30.681240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681165    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" containerName="maas-api"
Sep 10 23:51:30.681240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681179    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c51a1d-d12a-48da-a6ef-c88f5e897c05" containerName="maas-api"
Sep 10 23:51:30.681240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681189    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c51a1d-d12a-48da-a6ef-c88f5e897c05" containerName="maas-api"
Sep 10 23:51:30.681240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681219    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c28a885-e231-4304-8f01-c92a3d5fad15" containerName="payload-pre-processing"
Sep 10 23:51:30.681240 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681228    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c28a885-e231-4304-8f01-c92a3d5fad15" containerName="payload-pre-processing"
Sep 10 23:51:30.681619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681315    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1004c271-4d37-4265-ae31-9d53edcf3b8a" containerName="payload-processing"
Sep 10 23:51:30.681619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681328    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6b9ea8-7872-4d7d-9875-61a1a77ccdfd" containerName="maas-api"
Sep 10 23:51:30.681619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681338    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c28a885-e231-4304-8f01-c92a3d5fad15" containerName="payload-pre-processing"
Sep 10 23:51:30.681619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.681347    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c51a1d-d12a-48da-a6ef-c88f5e897c05" containerName="maas-api"
Sep 10 23:51:30.686614 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.686591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.689885 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.689862    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a4b2fb82-serving-cert\""
Sep 10 23:51:30.697278 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.697251    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"]
Sep 10 23:51:30.723594 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.723572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:30.785153 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.785129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-metrics-certs\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.785260 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.785181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-openshift-service-ca\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.785260 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.785208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7mbp\" (UniqueName: \"kubernetes.io/projected/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-kube-api-access-b7mbp\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.785374 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.785279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-maas-api-tls\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.825723 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.825696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:30.886035 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.886011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-openshift-service-ca\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.886126 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.886043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7mbp\" (UniqueName: \"kubernetes.io/projected/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-kube-api-access-b7mbp\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.886126 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.886075    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-maas-api-tls\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.886213 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.886132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-metrics-certs\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.886707 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.886679    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-openshift-service-ca\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.888504 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.888487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-maas-api-tls\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.888580 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.888516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-metrics-certs\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.893315 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.893298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7mbp\" (UniqueName: \"kubernetes.io/projected/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-kube-api-access-b7mbp\") pod \"maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:30.927202 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:30.927174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:31.001273 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.001252    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:31.205269 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.205160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:31.275194 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.275164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"]
Sep 10 23:51:31.277027 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:51:31.276997    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb38cff_d8e2_4f4b_b8b9_2c01604ad911.slice/crio-0ce5056ae5a9888c36886962986daf352e1c027c2fdbe4c0115a922c23a7305c WatchSource:0}: Error finding container 0ce5056ae5a9888c36886962986daf352e1c027c2fdbe4c0115a922c23a7305c: Status 404 returned error can't find the container with id 0ce5056ae5a9888c36886962986daf352e1c027c2fdbe4c0115a922c23a7305c
Sep 10 23:51:31.301651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.301627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:51:31.312853 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.312820    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"]
Sep 10 23:51:31.317293 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.317276    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:31.320244 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.320215    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a4b2fb82\""
Sep 10 23:51:31.320339 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.320271    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a4b2fb82-dockercfg-d9dpr\""
Sep 10 23:51:31.327134 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.327115    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"]
Sep 10 23:51:31.348376 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.348351    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"]
Sep 10 23:51:31.351683 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.351665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:31.362753 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.362730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"]
Sep 10 23:51:31.389691 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.389667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5534f-52cf-4aaf-a67f-0156523447ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl\" (UID: \"c4e5534f-52cf-4aaf-a67f-0156523447ca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:31.389821 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.389697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9zzw\" (UniqueName: \"kubernetes.io/projected/42b73d36-4dff-458b-92cc-c5f49d60b0c0-kube-api-access-k9zzw\") pod \"payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9\" (UID: \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\") " pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:31.389821 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.389718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b73d36-4dff-458b-92cc-c5f49d60b0c0-plugins-config-volume\") pod \"payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9\" (UID: \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\") " pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:31.389957 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.389850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzpq2\" (UniqueName: \"kubernetes.io/projected/c4e5534f-52cf-4aaf-a67f-0156523447ca-kube-api-access-qzpq2\") pod \"payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl\" (UID: \"c4e5534f-52cf-4aaf-a67f-0156523447ca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:31.491275 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.491235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b73d36-4dff-458b-92cc-c5f49d60b0c0-plugins-config-volume\") pod \"payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9\" (UID: \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\") " pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:31.491441 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.491365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzpq2\" (UniqueName: \"kubernetes.io/projected/c4e5534f-52cf-4aaf-a67f-0156523447ca-kube-api-access-qzpq2\") pod \"payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl\" (UID: \"c4e5534f-52cf-4aaf-a67f-0156523447ca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:31.491532 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.491456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5534f-52cf-4aaf-a67f-0156523447ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl\" (UID: \"c4e5534f-52cf-4aaf-a67f-0156523447ca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:31.491532 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.491474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9zzw\" (UniqueName: \"kubernetes.io/projected/42b73d36-4dff-458b-92cc-c5f49d60b0c0-kube-api-access-k9zzw\") pod \"payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9\" (UID: \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\") " pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:31.492051 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.492028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b73d36-4dff-458b-92cc-c5f49d60b0c0-plugins-config-volume\") pod \"payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9\" (UID: \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\") " pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:31.492196 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.492179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5534f-52cf-4aaf-a67f-0156523447ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl\" (UID: \"c4e5534f-52cf-4aaf-a67f-0156523447ca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:31.500602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.500581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzpq2\" (UniqueName: \"kubernetes.io/projected/c4e5534f-52cf-4aaf-a67f-0156523447ca-kube-api-access-qzpq2\") pod \"payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl\" (UID: \"c4e5534f-52cf-4aaf-a67f-0156523447ca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:31.500985 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.500967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9zzw\" (UniqueName: \"kubernetes.io/projected/42b73d36-4dff-458b-92cc-c5f49d60b0c0-kube-api-access-k9zzw\") pod \"payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9\" (UID: \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\") " pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:31.628037 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.627969    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:31.662516 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.662493    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:31.803503 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.803456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2" event={"ID":"adb38cff-d8e2-4f4b-b8b9-2c01604ad911","Type":"ContainerStarted","Data":"77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40"}
Sep 10 23:51:31.803976 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.803534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2" event={"ID":"adb38cff-d8e2-4f4b-b8b9-2c01604ad911","Type":"ContainerStarted","Data":"0ce5056ae5a9888c36886962986daf352e1c027c2fdbe4c0115a922c23a7305c"}
Sep 10 23:51:31.803976 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.803564    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:31.824673 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.824627    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2" podStartSLOduration=1.464490468 podStartE2EDuration="1.824613267s" totalImagesPullingTime="360.122799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:31.824211144 +0000 UTC m=+903.655238691" watchObservedRunningTime="2026-09-10 23:51:31.824613267 +0000 UTC m=+903.655640824"
Sep 10 23:51:31.911621 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.911591    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"]
Sep 10 23:51:31.914165 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:51:31.914141    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e5534f_52cf_4aaf_a67f_0156523447ca.slice/crio-121e71335a746d001c33d91d25314ce496b54ebd4d9f74ea2066fed3ab1a7215 WatchSource:0}: Error finding container 121e71335a746d001c33d91d25314ce496b54ebd4d9f74ea2066fed3ab1a7215: Status 404 returned error can't find the container with id 121e71335a746d001c33d91d25314ce496b54ebd4d9f74ea2066fed3ab1a7215
Sep 10 23:51:31.954016 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:31.953982    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"]
Sep 10 23:51:31.955719 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:51:31.955699    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b73d36_4dff_458b_92cc_c5f49d60b0c0.slice/crio-387762a07044089cc0a00866c1a698dba53046a1841951cd9c30cde62d35e134 WatchSource:0}: Error finding container 387762a07044089cc0a00866c1a698dba53046a1841951cd9c30cde62d35e134: Status 404 returned error can't find the container with id 387762a07044089cc0a00866c1a698dba53046a1841951cd9c30cde62d35e134
Sep 10 23:51:32.810650 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:32.810611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl" event={"ID":"c4e5534f-52cf-4aaf-a67f-0156523447ca","Type":"ContainerStarted","Data":"5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd"}
Sep 10 23:51:32.811132 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:32.810682    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:51:32.811132 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:32.810707    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl" event={"ID":"c4e5534f-52cf-4aaf-a67f-0156523447ca","Type":"ContainerStarted","Data":"121e71335a746d001c33d91d25314ce496b54ebd4d9f74ea2066fed3ab1a7215"}
Sep 10 23:51:32.813171 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:32.813142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9" event={"ID":"42b73d36-4dff-458b-92cc-c5f49d60b0c0","Type":"ContainerStarted","Data":"de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b"}
Sep 10 23:51:32.813281 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:32.813193    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:32.813281 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:32.813216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9" event={"ID":"42b73d36-4dff-458b-92cc-c5f49d60b0c0","Type":"ContainerStarted","Data":"387762a07044089cc0a00866c1a698dba53046a1841951cd9c30cde62d35e134"}
Sep 10 23:51:32.827613 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:32.827574    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl" podStartSLOduration=1.827559417 podStartE2EDuration="1.827559417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:32.825498756 +0000 UTC m=+904.656526343" watchObservedRunningTime="2026-09-10 23:51:32.827559417 +0000 UTC m=+904.658586949"
Sep 10 23:51:32.841523 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:32.841483    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9" podStartSLOduration=1.8414679189999998 podStartE2EDuration="1.841467919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:32.84112173 +0000 UTC m=+904.672149282" watchObservedRunningTime="2026-09-10 23:51:32.841467919 +0000 UTC m=+904.672495452"
Sep 10 23:51:37.817996 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:37.817967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:51:43.819672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:43.819641    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:51:43.820051 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:51:43.819804    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:52:01.243874 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.243842    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"]
Sep 10 23:52:01.246549 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.244094    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2" podUID="adb38cff-d8e2-4f4b-b8b9-2c01604ad911" containerName="maas-api" containerID="cri-o://77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40" gracePeriod=30
Sep 10 23:52:01.274791 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.274765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:01.299086 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.299049    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"]
Sep 10 23:52:01.299308 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.299281    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9" podUID="42b73d36-4dff-458b-92cc-c5f49d60b0c0" containerName="payload-processing" containerID="cri-o://de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b" gracePeriod=30
Sep 10 23:52:01.304548 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.304521    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"]
Sep 10 23:52:01.304876 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.304850    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl" podUID="c4e5534f-52cf-4aaf-a67f-0156523447ca" containerName="payload-pre-processing" containerID="cri-o://5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd" gracePeriod=30
Sep 10 23:52:01.565472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.565341    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:52:01.610885 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.610859    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:52:01.616061 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.615979    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:52:01.638414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.638393    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7mbp\" (UniqueName: \"kubernetes.io/projected/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-kube-api-access-b7mbp\") pod \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") "
Sep 10 23:52:01.638556 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.638421    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-maas-api-tls\") pod \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") "
Sep 10 23:52:01.638556 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.638443    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-openshift-service-ca\") pod \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") "
Sep 10 23:52:01.638556 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.638490    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-metrics-certs\") pod \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\" (UID: \"adb38cff-d8e2-4f4b-b8b9-2c01604ad911\") "
Sep 10 23:52:01.638895 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.638860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adb38cff-d8e2-4f4b-b8b9-2c01604ad911" (UID: "adb38cff-d8e2-4f4b-b8b9-2c01604ad911"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:52:01.640623 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.640601    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adb38cff-d8e2-4f4b-b8b9-2c01604ad911" (UID: "adb38cff-d8e2-4f4b-b8b9-2c01604ad911"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:52:01.641043 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.641023    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "adb38cff-d8e2-4f4b-b8b9-2c01604ad911" (UID: "adb38cff-d8e2-4f4b-b8b9-2c01604ad911"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:52:01.641107 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.641077    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-kube-api-access-b7mbp" (OuterVolumeSpecName: "kube-api-access-b7mbp") pod "adb38cff-d8e2-4f4b-b8b9-2c01604ad911" (UID: "adb38cff-d8e2-4f4b-b8b9-2c01604ad911"). InnerVolumeSpecName "kube-api-access-b7mbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:52:01.739146 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739117    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5534f-52cf-4aaf-a67f-0156523447ca-plugins-config-volume\") pod \"c4e5534f-52cf-4aaf-a67f-0156523447ca\" (UID: \"c4e5534f-52cf-4aaf-a67f-0156523447ca\") "
Sep 10 23:52:01.739146 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739151    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzpq2\" (UniqueName: \"kubernetes.io/projected/c4e5534f-52cf-4aaf-a67f-0156523447ca-kube-api-access-qzpq2\") pod \"c4e5534f-52cf-4aaf-a67f-0156523447ca\" (UID: \"c4e5534f-52cf-4aaf-a67f-0156523447ca\") "
Sep 10 23:52:01.739366 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b73d36-4dff-458b-92cc-c5f49d60b0c0-plugins-config-volume\") pod \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\" (UID: \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\") "
Sep 10 23:52:01.739423 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9zzw\" (UniqueName: \"kubernetes.io/projected/42b73d36-4dff-458b-92cc-c5f49d60b0c0-kube-api-access-k9zzw\") pod \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\" (UID: \"42b73d36-4dff-458b-92cc-c5f49d60b0c0\") "
Sep 10 23:52:01.739547 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739526    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e5534f-52cf-4aaf-a67f-0156523447ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4e5534f-52cf-4aaf-a67f-0156523447ca" (UID: "c4e5534f-52cf-4aaf-a67f-0156523447ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:52:01.739621 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b73d36-4dff-458b-92cc-c5f49d60b0c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42b73d36-4dff-458b-92cc-c5f49d60b0c0" (UID: "42b73d36-4dff-458b-92cc-c5f49d60b0c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:52:01.739885 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739861    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7mbp\" (UniqueName: \"kubernetes.io/projected/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-kube-api-access-b7mbp\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:01.739885 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739886    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:01.740133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739901    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b73d36-4dff-458b-92cc-c5f49d60b0c0-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:01.740133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739915    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:01.740133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739953    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb38cff-d8e2-4f4b-b8b9-2c01604ad911-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:01.740133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.739968    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5534f-52cf-4aaf-a67f-0156523447ca-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:01.741295 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.741275    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e5534f-52cf-4aaf-a67f-0156523447ca-kube-api-access-qzpq2" (OuterVolumeSpecName: "kube-api-access-qzpq2") pod "c4e5534f-52cf-4aaf-a67f-0156523447ca" (UID: "c4e5534f-52cf-4aaf-a67f-0156523447ca"). InnerVolumeSpecName "kube-api-access-qzpq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:52:01.741390 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.741346    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b73d36-4dff-458b-92cc-c5f49d60b0c0-kube-api-access-k9zzw" (OuterVolumeSpecName: "kube-api-access-k9zzw") pod "42b73d36-4dff-458b-92cc-c5f49d60b0c0" (UID: "42b73d36-4dff-458b-92cc-c5f49d60b0c0"). InnerVolumeSpecName "kube-api-access-k9zzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:52:01.840586 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.840507    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzpq2\" (UniqueName: \"kubernetes.io/projected/c4e5534f-52cf-4aaf-a67f-0156523447ca-kube-api-access-qzpq2\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:01.840586 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.840538    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9zzw\" (UniqueName: \"kubernetes.io/projected/42b73d36-4dff-458b-92cc-c5f49d60b0c0-kube-api-access-k9zzw\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:01.948977 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.948913    2562 generic.go:356] "Generic (PLEG): container finished" podID="c4e5534f-52cf-4aaf-a67f-0156523447ca" containerID="5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd" exitCode=0
Sep 10 23:52:01.948977 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.948971    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"
Sep 10 23:52:01.949219 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.949017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl" event={"ID":"c4e5534f-52cf-4aaf-a67f-0156523447ca","Type":"ContainerDied","Data":"5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd"}
Sep 10 23:52:01.949219 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.949073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl" event={"ID":"c4e5534f-52cf-4aaf-a67f-0156523447ca","Type":"ContainerDied","Data":"121e71335a746d001c33d91d25314ce496b54ebd4d9f74ea2066fed3ab1a7215"}
Sep 10 23:52:01.949219 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.949095    2562 scope.go:117] "RemoveContainer" containerID="5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd"
Sep 10 23:52:01.950361 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.950338    2562 generic.go:356] "Generic (PLEG): container finished" podID="adb38cff-d8e2-4f4b-b8b9-2c01604ad911" containerID="77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40" exitCode=0
Sep 10 23:52:01.950450 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.950383    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"
Sep 10 23:52:01.950450 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.950434    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2" event={"ID":"adb38cff-d8e2-4f4b-b8b9-2c01604ad911","Type":"ContainerDied","Data":"77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40"}
Sep 10 23:52:01.950564 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.950488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2" event={"ID":"adb38cff-d8e2-4f4b-b8b9-2c01604ad911","Type":"ContainerDied","Data":"0ce5056ae5a9888c36886962986daf352e1c027c2fdbe4c0115a922c23a7305c"}
Sep 10 23:52:01.951987 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.951948    2562 generic.go:356] "Generic (PLEG): container finished" podID="42b73d36-4dff-458b-92cc-c5f49d60b0c0" containerID="de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b" exitCode=0
Sep 10 23:52:01.952214 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.952006    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"
Sep 10 23:52:01.952214 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.952005    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9" event={"ID":"42b73d36-4dff-458b-92cc-c5f49d60b0c0","Type":"ContainerDied","Data":"de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b"}
Sep 10 23:52:01.952214 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.952148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9" event={"ID":"42b73d36-4dff-458b-92cc-c5f49d60b0c0","Type":"ContainerDied","Data":"387762a07044089cc0a00866c1a698dba53046a1841951cd9c30cde62d35e134"}
Sep 10 23:52:01.959195 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.959174    2562 scope.go:117] "RemoveContainer" containerID="5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd"
Sep 10 23:52:01.959483 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:52:01.959463    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd\": container with ID starting with 5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd not found: ID does not exist" containerID="5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd"
Sep 10 23:52:01.959583 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.959491    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd"} err="failed to get container status \"5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd\": rpc error: code = NotFound desc = could not find container \"5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd\": container with ID starting with 5389e1bfba94bef9611a0b92e8df9974e0508c3a2fa3d9b14e5a78add04e61dd not found: ID does not exist"
Sep 10 23:52:01.959583 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.959513    2562 scope.go:117] "RemoveContainer" containerID="77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40"
Sep 10 23:52:01.967619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.967601    2562 scope.go:117] "RemoveContainer" containerID="77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40"
Sep 10 23:52:01.967871 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:52:01.967852    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40\": container with ID starting with 77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40 not found: ID does not exist" containerID="77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40"
Sep 10 23:52:01.968042 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.967881    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40"} err="failed to get container status \"77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40\": rpc error: code = NotFound desc = could not find container \"77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40\": container with ID starting with 77d7c8240c57a11b45fd407c8d6acbc86c8ca25a6544b43f52d1839feb677f40 not found: ID does not exist"
Sep 10 23:52:01.968042 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.967904    2562 scope.go:117] "RemoveContainer" containerID="de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b"
Sep 10 23:52:01.972656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.972636    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"]
Sep 10 23:52:01.975615 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.975593    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a4b2fb82-56bdcfcbdb-xdwhl"]
Sep 10 23:52:01.976739 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.976722    2562 scope.go:117] "RemoveContainer" containerID="de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b"
Sep 10 23:52:01.977099 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:52:01.977081    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b\": container with ID starting with de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b not found: ID does not exist" containerID="de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b"
Sep 10 23:52:01.977157 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.977105    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b"} err="failed to get container status \"de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b\": rpc error: code = NotFound desc = could not find container \"de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b\": container with ID starting with de1491f97fb19bbcf036a5f4e13269f3daf60cdda8ff4ee3852703e72dcc9a8b not found: ID does not exist"
Sep 10 23:52:01.986128 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.986109    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"]
Sep 10 23:52:01.989651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:01.989632    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a4b2fb82-659c869977-mjbr9"]
Sep 10 23:52:02.000982 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:02.000962    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"]
Sep 10 23:52:02.005258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:02.005238    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a4b2fb82-5d7db94495-jchc2"]
Sep 10 23:52:02.655317 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:02.655246    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b73d36-4dff-458b-92cc-c5f49d60b0c0" path="/var/lib/kubelet/pods/42b73d36-4dff-458b-92cc-c5f49d60b0c0/volumes"
Sep 10 23:52:02.655667 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:02.655655    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb38cff-d8e2-4f4b-b8b9-2c01604ad911" path="/var/lib/kubelet/pods/adb38cff-d8e2-4f4b-b8b9-2c01604ad911/volumes"
Sep 10 23:52:02.656030 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:02.656015    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e5534f-52cf-4aaf-a67f-0156523447ca" path="/var/lib/kubelet/pods/c4e5534f-52cf-4aaf-a67f-0156523447ca/volumes"
Sep 10 23:52:02.891999 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:02.891955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:03.803050 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:03.803007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:05.146330 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:05.146295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:05.978675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:05.978639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:09.724653 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:09.724621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:09.830490 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:09.830455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:09.972130 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:09.972100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:10.079068 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:10.078994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:10.385985 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:10.385900    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:11.032992 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.032958    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"]
Sep 10 23:52:11.033384 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033335    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb38cff-d8e2-4f4b-b8b9-2c01604ad911" containerName="maas-api"
Sep 10 23:52:11.033384 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033347    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb38cff-d8e2-4f4b-b8b9-2c01604ad911" containerName="maas-api"
Sep 10 23:52:11.033384 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033367    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b73d36-4dff-458b-92cc-c5f49d60b0c0" containerName="payload-processing"
Sep 10 23:52:11.033384 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033373    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b73d36-4dff-458b-92cc-c5f49d60b0c0" containerName="payload-processing"
Sep 10 23:52:11.033384 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033380    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e5534f-52cf-4aaf-a67f-0156523447ca" containerName="payload-pre-processing"
Sep 10 23:52:11.033384 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033387    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e5534f-52cf-4aaf-a67f-0156523447ca" containerName="payload-pre-processing"
Sep 10 23:52:11.033606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033452    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e5534f-52cf-4aaf-a67f-0156523447ca" containerName="payload-pre-processing"
Sep 10 23:52:11.033606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033463    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb38cff-d8e2-4f4b-b8b9-2c01604ad911" containerName="maas-api"
Sep 10 23:52:11.033606 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.033471    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b73d36-4dff-458b-92cc-c5f49d60b0c0" containerName="payload-processing"
Sep 10 23:52:11.037895 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.037874    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.040756 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.040735    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-e6d58266-kserve-self-signed-certs\""
Sep 10 23:52:11.051652 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.051622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"]
Sep 10 23:52:11.065655 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.065629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:11.115675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.115645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-home\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.115794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.115690    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56wkr\" (UniqueName: \"kubernetes.io/projected/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-kube-api-access-56wkr\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.115794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.115715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tmp-dir\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.115794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.115762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tls-certs\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.115956 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.115802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-dshm\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.115956 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.115841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-model-cache\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.216531 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.216506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56wkr\" (UniqueName: \"kubernetes.io/projected/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-kube-api-access-56wkr\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.216656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.216541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tmp-dir\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.216656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.216558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tls-certs\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.216656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.216595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-dshm\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.216656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.216608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-model-cache\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.216656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.216649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-home\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.217000 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.216979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tmp-dir\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.217094 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.217072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-model-cache\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.217145 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.217093    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-home\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.218891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.218874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-dshm\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.219090 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.219065    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tls-certs\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.224583 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.224562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56wkr\" (UniqueName: \"kubernetes.io/projected/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-kube-api-access-56wkr\") pod \"e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") " pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.347193 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.347128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:11.620864 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.620839    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"]
Sep 10 23:52:11.623289 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:52:11.623263    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d5aa267_f277_44ce_bd7d_ee2c4d7df4e3.slice/crio-87f6654d907b85db63e284780c6b47ff838483a2f8c0e1baa5273a673e27862e WatchSource:0}: Error finding container 87f6654d907b85db63e284780c6b47ff838483a2f8c0e1baa5273a673e27862e: Status 404 returned error can't find the container with id 87f6654d907b85db63e284780c6b47ff838483a2f8c0e1baa5273a673e27862e
Sep 10 23:52:11.999833 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.999797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s" event={"ID":"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3","Type":"ContainerStarted","Data":"7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65"}
Sep 10 23:52:11.999993 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:11.999845    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s" event={"ID":"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3","Type":"ContainerStarted","Data":"87f6654d907b85db63e284780c6b47ff838483a2f8c0e1baa5273a673e27862e"}
Sep 10 23:52:12.018264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:12.018215    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s" podStartSLOduration=1.018201089 podStartE2EDuration="1.018201089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:12.016432693 +0000 UTC m=+943.847460247" watchObservedRunningTime="2026-09-10 23:52:12.018201089 +0000 UTC m=+943.849228666"
Sep 10 23:52:21.348242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:21.348208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:21.348607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:21.348264    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:21.360911 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:21.360884    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:22.056143 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:22.056111    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:33.229861 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.229822    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"]
Sep 10 23:52:33.235891 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.235870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.239806 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.239689    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1fe290\"/\"openshift-service-ca.crt\""
Sep 10 23:52:33.239806 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.239710    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1fe290\"/\"kube-root-ca.crt\""
Sep 10 23:52:33.240043 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.240021    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1fe290\"/\"test-model-1fe290-kserve-self-signed-certs\""
Sep 10 23:52:33.240043 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.240039    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1fe290\"/\"default-dockercfg-vvwws\""
Sep 10 23:52:33.241880 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.241853    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"]
Sep 10 23:52:33.256574 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.256550    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:33.291894 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.291867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-dshm\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.292024 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.291944    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da23751-134e-4c4d-9527-fe292c212d2e-tls-certs\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.292098 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.292031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-tmp-dir\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.292098 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.292073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplqc\" (UniqueName: \"kubernetes.io/projected/2da23751-134e-4c4d-9527-fe292c212d2e-kube-api-access-wplqc\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.292209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.292110    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-home\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.292209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.292168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-model-cache\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.393370 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.393344    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-dshm\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.393460 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.393393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da23751-134e-4c4d-9527-fe292c212d2e-tls-certs\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.393460 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.393430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-tmp-dir\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.393550 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.393530    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplqc\" (UniqueName: \"kubernetes.io/projected/2da23751-134e-4c4d-9527-fe292c212d2e-kube-api-access-wplqc\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.393781 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.393582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-home\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.394242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.394037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-model-cache\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.394242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.394109    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-tmp-dir\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.394242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.394174    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-home\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.394435 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.394315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-model-cache\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.399710 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.396701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-dshm\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.399710 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.396716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da23751-134e-4c4d-9527-fe292c212d2e-tls-certs\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.403827 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.403804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplqc\" (UniqueName: \"kubernetes.io/projected/2da23751-134e-4c4d-9527-fe292c212d2e-kube-api-access-wplqc\") pod \"test-model-1fe290-kserve-695c9666cf-8mpzn\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") " pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.546383 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.546316    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:33.818824 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:33.818750    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"]
Sep 10 23:52:33.821724 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:52:33.821694    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2da23751_134e_4c4d_9527_fe292c212d2e.slice/crio-d64a5fd23582db72dc4272a70684dd83f1d5c3c681743bd8ea5ddfa22259383f WatchSource:0}: Error finding container d64a5fd23582db72dc4272a70684dd83f1d5c3c681743bd8ea5ddfa22259383f: Status 404 returned error can't find the container with id d64a5fd23582db72dc4272a70684dd83f1d5c3c681743bd8ea5ddfa22259383f
Sep 10 23:52:34.103409 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:34.103334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn" event={"ID":"2da23751-134e-4c4d-9527-fe292c212d2e","Type":"ContainerStarted","Data":"07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61"}
Sep 10 23:52:34.103409 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:34.103371    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn" event={"ID":"2da23751-134e-4c4d-9527-fe292c212d2e","Type":"ContainerStarted","Data":"d64a5fd23582db72dc4272a70684dd83f1d5c3c681743bd8ea5ddfa22259383f"}
Sep 10 23:52:34.122464 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:34.122416    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn" podStartSLOduration=1.122401574 podStartE2EDuration="1.122401574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:34.119778745 +0000 UTC m=+965.950806297" watchObservedRunningTime="2026-09-10 23:52:34.122401574 +0000 UTC m=+965.953429107"
Sep 10 23:52:36.116168 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:36.116133    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"]
Sep 10 23:52:36.129754 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:36.129720    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:36.335545 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:36.335480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:36.819071 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:36.819037    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"]
Sep 10 23:52:36.842194 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:36.842159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:38.778838 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:38.778807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:38.853560 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:38.853528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:39.030167 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:39.030083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:39.440213 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:39.440186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:39.778568 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:39.778483    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:40.243106 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:40.243065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:42.300747 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:42.300713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:42.404299 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:42.404265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:42.520811 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:42.520779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:42.634655 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:42.634579    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:42.746980 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:42.746941    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:43.547026 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.546989    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:43.547402 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.547039    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:43.559644 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.559620    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:43.760464 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.760435    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"]
Sep 10 23:52:43.765195 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.765174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.770834 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.770815    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b88386-serving-cert\""
Sep 10 23:52:43.778014 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.777904    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"]
Sep 10 23:52:43.803560 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.803500    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:43.882248 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.882222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwt9k\" (UniqueName: \"kubernetes.io/projected/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-kube-api-access-nwt9k\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.882413 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.882256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-maas-api-tls\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.882413 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.882344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-openshift-service-ca\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.882514 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.882407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-metrics-certs\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.903378 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.903344    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:43.983806 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.983778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwt9k\" (UniqueName: \"kubernetes.io/projected/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-kube-api-access-nwt9k\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.983952 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.983817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-maas-api-tls\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.983952 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.983865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-openshift-service-ca\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.983952 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.983900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-metrics-certs\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.984605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.984584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-openshift-service-ca\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.986265 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.986239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-maas-api-tls\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.986358 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.986298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-metrics-certs\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:43.992820 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:43.992791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwt9k\" (UniqueName: \"kubernetes.io/projected/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-kube-api-access-nwt9k\") pod \"maas-api-e2e-explicit-b88386-f9fd4d766-67scn\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:44.004016 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:44.003996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:44.076599 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:44.076549    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:44.164534 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:44.164506    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:52:44.354869 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:44.354842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"]
Sep 10 23:52:44.357048 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:52:44.357006    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a868a9_6c4f_4a39_a7db_fe3cf39d10a5.slice/crio-bbbea8006a74eeadad1aafb37dda7a85e1e9b34f6dd69f31f783f1b4513a11e6 WatchSource:0}: Error finding container bbbea8006a74eeadad1aafb37dda7a85e1e9b34f6dd69f31f783f1b4513a11e6: Status 404 returned error can't find the container with id bbbea8006a74eeadad1aafb37dda7a85e1e9b34f6dd69f31f783f1b4513a11e6
Sep 10 23:52:44.358486 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:44.358466    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:52:44.398255 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:44.398226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:44.563689 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:44.563654    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:45.164264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:45.164228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn" event={"ID":"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5","Type":"ContainerStarted","Data":"ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8"}
Sep 10 23:52:45.164426 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:45.164405    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:45.164478 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:45.164439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn" event={"ID":"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5","Type":"ContainerStarted","Data":"bbbea8006a74eeadad1aafb37dda7a85e1e9b34f6dd69f31f783f1b4513a11e6"}
Sep 10 23:52:45.215156 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:45.215111    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn" podStartSLOduration=1.6037007380000001 podStartE2EDuration="2.215095111s" totalImagesPullingTime="611.394373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:45.212798795 +0000 UTC m=+977.043826347" watchObservedRunningTime="2026-09-10 23:52:45.215095111 +0000 UTC m=+977.046122638"
Sep 10 23:52:51.136380 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.136296    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn" podUID="cecd6744-7e3e-4077-81f8-1c9a82d55527" containerName="main" containerID="cri-o://0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1" gracePeriod=45
Sep 10 23:52:51.174372 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.174351    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:52:51.384966 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.384936    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:52:51.448739 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.448714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54nl2\" (UniqueName: \"kubernetes.io/projected/cecd6744-7e3e-4077-81f8-1c9a82d55527-kube-api-access-54nl2\") pod \"cecd6744-7e3e-4077-81f8-1c9a82d55527\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") "
Sep 10 23:52:51.448844 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.448810    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-tmp-dir\") pod \"cecd6744-7e3e-4077-81f8-1c9a82d55527\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") "
Sep 10 23:52:51.448844 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.448835    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6744-7e3e-4077-81f8-1c9a82d55527-tls-certs\") pod \"cecd6744-7e3e-4077-81f8-1c9a82d55527\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") "
Sep 10 23:52:51.448986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.448858    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-home\") pod \"cecd6744-7e3e-4077-81f8-1c9a82d55527\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") "
Sep 10 23:52:51.448986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.448876    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-model-cache\") pod \"cecd6744-7e3e-4077-81f8-1c9a82d55527\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") "
Sep 10 23:52:51.448986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.448895    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-dshm\") pod \"cecd6744-7e3e-4077-81f8-1c9a82d55527\" (UID: \"cecd6744-7e3e-4077-81f8-1c9a82d55527\") "
Sep 10 23:52:51.449146 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.449108    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cecd6744-7e3e-4077-81f8-1c9a82d55527" (UID: "cecd6744-7e3e-4077-81f8-1c9a82d55527"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:52:51.449146 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.449123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-home" (OuterVolumeSpecName: "home") pod "cecd6744-7e3e-4077-81f8-1c9a82d55527" (UID: "cecd6744-7e3e-4077-81f8-1c9a82d55527"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:52:51.449258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.449241    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:51.449315 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.449250    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-model-cache" (OuterVolumeSpecName: "model-cache") pod "cecd6744-7e3e-4077-81f8-1c9a82d55527" (UID: "cecd6744-7e3e-4077-81f8-1c9a82d55527"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:52:51.449315 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.449261    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:51.450959 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.450912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-dshm" (OuterVolumeSpecName: "dshm") pod "cecd6744-7e3e-4077-81f8-1c9a82d55527" (UID: "cecd6744-7e3e-4077-81f8-1c9a82d55527"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:52:51.451054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.450985    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecd6744-7e3e-4077-81f8-1c9a82d55527-kube-api-access-54nl2" (OuterVolumeSpecName: "kube-api-access-54nl2") pod "cecd6744-7e3e-4077-81f8-1c9a82d55527" (UID: "cecd6744-7e3e-4077-81f8-1c9a82d55527"). InnerVolumeSpecName "kube-api-access-54nl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:52:51.451054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.450987    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cecd6744-7e3e-4077-81f8-1c9a82d55527-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cecd6744-7e3e-4077-81f8-1c9a82d55527" (UID: "cecd6744-7e3e-4077-81f8-1c9a82d55527"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:52:51.550258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.550234    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54nl2\" (UniqueName: \"kubernetes.io/projected/cecd6744-7e3e-4077-81f8-1c9a82d55527-kube-api-access-54nl2\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:51.550258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.550259    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6744-7e3e-4077-81f8-1c9a82d55527-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:51.550399 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.550274    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:51.550399 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.550294    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6744-7e3e-4077-81f8-1c9a82d55527-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:51.838516 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:51.838444    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s" podUID="3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" containerName="main" containerID="cri-o://7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65" gracePeriod=45
Sep 10 23:52:52.074323 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.074298    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:52.155298 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155268    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-home\") pod \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") "
Sep 10 23:52:52.155669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155324    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tls-certs\") pod \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") "
Sep 10 23:52:52.155669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155353    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tmp-dir\") pod \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") "
Sep 10 23:52:52.155669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155374    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-dshm\") pod \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") "
Sep 10 23:52:52.155669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155403    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56wkr\" (UniqueName: \"kubernetes.io/projected/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-kube-api-access-56wkr\") pod \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") "
Sep 10 23:52:52.155669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155491    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-model-cache\") pod \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\" (UID: \"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3\") "
Sep 10 23:52:52.155669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155625    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-home" (OuterVolumeSpecName: "home") pod "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" (UID: "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:52:52.155909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" (UID: "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:52:52.155909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" (UID: "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:52:52.155909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155860    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.155909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155877    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.155909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.155887    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.157414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.157395    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-dshm" (OuterVolumeSpecName: "dshm") pod "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" (UID: "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:52:52.157909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.157885    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" (UID: "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:52:52.157909 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.157888    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-kube-api-access-56wkr" (OuterVolumeSpecName: "kube-api-access-56wkr") pod "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" (UID: "3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3"). InnerVolumeSpecName "kube-api-access-56wkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:52:52.199439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.199412    2562 generic.go:356] "Generic (PLEG): container finished" podID="cecd6744-7e3e-4077-81f8-1c9a82d55527" containerID="0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1" exitCode=0
Sep 10 23:52:52.199516 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.199452    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"
Sep 10 23:52:52.199516 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.199503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn" event={"ID":"cecd6744-7e3e-4077-81f8-1c9a82d55527","Type":"ContainerDied","Data":"0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1"}
Sep 10 23:52:52.199608 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.199544    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn" event={"ID":"cecd6744-7e3e-4077-81f8-1c9a82d55527","Type":"ContainerDied","Data":"eda09c97189737d551f7ce53cc8792f705be04bf788ce510929afbb32e4a8ece"}
Sep 10 23:52:52.199608 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.199562    2562 scope.go:117] "RemoveContainer" containerID="0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1"
Sep 10 23:52:52.201016 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.200996    2562 generic.go:356] "Generic (PLEG): container finished" podID="3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" containerID="7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65" exitCode=0
Sep 10 23:52:52.201096 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.201024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s" event={"ID":"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3","Type":"ContainerDied","Data":"7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65"}
Sep 10 23:52:52.201096 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.201046    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"
Sep 10 23:52:52.201096 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.201057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s" event={"ID":"3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3","Type":"ContainerDied","Data":"87f6654d907b85db63e284780c6b47ff838483a2f8c0e1baa5273a673e27862e"}
Sep 10 23:52:52.211563 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.211540    2562 scope.go:117] "RemoveContainer" containerID="0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1"
Sep 10 23:52:52.211838 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:52:52.211818    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1\": container with ID starting with 0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1 not found: ID does not exist" containerID="0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1"
Sep 10 23:52:52.211913 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.211845    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1"} err="failed to get container status \"0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1\": rpc error: code = NotFound desc = could not find container \"0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1\": container with ID starting with 0321167b0263f2196c9a90060e052556aa4133e0cc0bdc74f6c1c1a83f9c4fc1 not found: ID does not exist"
Sep 10 23:52:52.211913 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.211862    2562 scope.go:117] "RemoveContainer" containerID="7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65"
Sep 10 23:52:52.219324 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.219308    2562 scope.go:117] "RemoveContainer" containerID="7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65"
Sep 10 23:52:52.219596 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:52:52.219578    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65\": container with ID starting with 7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65 not found: ID does not exist" containerID="7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65"
Sep 10 23:52:52.219648 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.219603    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65"} err="failed to get container status \"7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65\": rpc error: code = NotFound desc = could not find container \"7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65\": container with ID starting with 7bfd7ffe5871dc4539fc8e1f14575e351f9f62380ebd2b8dd61c57e54d497c65 not found: ID does not exist"
Sep 10 23:52:52.230020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.229994    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"]
Sep 10 23:52:52.232988 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.232969    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-e6d58266-kserve-747bbcfd87-spnnn"]
Sep 10 23:52:52.245130 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.245106    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"]
Sep 10 23:52:52.251006 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.250985    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s"]
Sep 10 23:52:52.256635 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.256617    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.256693 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.256639    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.256693 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.256647    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56wkr\" (UniqueName: \"kubernetes.io/projected/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3-kube-api-access-56wkr\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:52:52.656046 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.656013    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" path="/var/lib/kubelet/pods/3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3/volumes"
Sep 10 23:52:52.656411 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:52.656398    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cecd6744-7e3e-4077-81f8-1c9a82d55527" path="/var/lib/kubelet/pods/cecd6744-7e3e-4077-81f8-1c9a82d55527/volumes"
Sep 10 23:52:57.046228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:57.046185    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-conflict-b-e6d58266-kserve-78cc448989-r7g6s" podUID="3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 10 23:52:58.345101 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.345065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:58.908795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.908766    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"]
Sep 10 23:52:58.909154 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.909142    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" containerName="main"
Sep 10 23:52:58.909205 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.909156    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" containerName="main"
Sep 10 23:52:58.909205 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.909169    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cecd6744-7e3e-4077-81f8-1c9a82d55527" containerName="main"
Sep 10 23:52:58.909205 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.909175    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cecd6744-7e3e-4077-81f8-1c9a82d55527" containerName="main"
Sep 10 23:52:58.909300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.909238    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d5aa267-f277-44ce-bd7d-ee2c4d7df4e3" containerName="main"
Sep 10 23:52:58.909300 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.909249    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cecd6744-7e3e-4077-81f8-1c9a82d55527" containerName="main"
Sep 10 23:52:58.912542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.912520    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:58.915611 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.915593    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7f598e\"/\"openshift-service-ca.crt\""
Sep 10 23:52:58.915857 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.915842    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7f598e\"/\"kube-root-ca.crt\""
Sep 10 23:52:58.915957 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.915892    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7f598e\"/\"test-model-7f598e-kserve-self-signed-certs\""
Sep 10 23:52:58.916619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.916602    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7f598e\"/\"default-dockercfg-qm5bj\""
Sep 10 23:52:58.924638 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:58.924617    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"]
Sep 10 23:52:59.011659 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.011620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-model-cache\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.011793 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.011666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tmp-dir\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.011793 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.011710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tls-certs\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.011793 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.011768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-dshm\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.011922 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.011798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gknzl\" (UniqueName: \"kubernetes.io/projected/f5925694-c4df-4f80-a1fb-c9a44472fc7f-kube-api-access-gknzl\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.011922 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.011903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-home\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.112469 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.112444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-model-cache\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.112596 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.112475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tmp-dir\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.112596 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.112493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tls-certs\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.112596 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.112536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-dshm\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.112596 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.112566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gknzl\" (UniqueName: \"kubernetes.io/projected/f5925694-c4df-4f80-a1fb-c9a44472fc7f-kube-api-access-gknzl\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.112816 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.112638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-home\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.112958 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.112912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-model-cache\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.113121 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.113062    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-home\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.113204 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.113171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tmp-dir\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.115225 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.115197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-dshm\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.115486 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.115466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tls-certs\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.125816 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.125794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gknzl\" (UniqueName: \"kubernetes.io/projected/f5925694-c4df-4f80-a1fb-c9a44472fc7f-kube-api-access-gknzl\") pod \"test-model-7f598e-kserve-6d9b86c4cc-q7pp9\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") " pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.157556 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.157512    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:52:59.221711 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.221687    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:52:59.491693 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:52:59.491619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"]
Sep 10 23:52:59.494435 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:52:59.494410    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5925694_c4df_4f80_a1fb_c9a44472fc7f.slice/crio-7da9330a49f68c07edcd7a664fde67281286f36283317e7a50589c6661af6031 WatchSource:0}: Error finding container 7da9330a49f68c07edcd7a664fde67281286f36283317e7a50589c6661af6031: Status 404 returned error can't find the container with id 7da9330a49f68c07edcd7a664fde67281286f36283317e7a50589c6661af6031
Sep 10 23:53:00.244093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:00.244051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9" event={"ID":"f5925694-c4df-4f80-a1fb-c9a44472fc7f","Type":"ContainerStarted","Data":"426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea"}
Sep 10 23:53:00.244093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:00.244093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9" event={"ID":"f5925694-c4df-4f80-a1fb-c9a44472fc7f","Type":"ContainerStarted","Data":"7da9330a49f68c07edcd7a664fde67281286f36283317e7a50589c6661af6031"}
Sep 10 23:53:00.262699 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:00.262648    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9" podStartSLOduration=2.262634425 podStartE2EDuration="2.262634425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:00.261595061 +0000 UTC m=+992.092622606" watchObservedRunningTime="2026-09-10 23:53:00.262634425 +0000 UTC m=+992.093661957"
Sep 10 23:53:04.389209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:04.389136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:04.493052 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:04.493022    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:04.598274 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:04.598242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:07.514603 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:07.514568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:09.222459 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:09.222425    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:53:09.222459 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:09.222465    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:53:09.234903 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:09.234878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:53:09.296258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:09.296231    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:53:15.069103 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:15.069075    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:15.808161 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:15.808132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:21.954794 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:21.954763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:22.064943 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:22.064883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:22.211133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:22.211043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:22.321115 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:22.321080    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:22.934852 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:22.934814    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:23.023417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:23.023385    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:23.122883 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:23.122848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:24.560385 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:24.560351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:30.539203 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:30.539167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:30.608095 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:30.608065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:31.286339 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:31.286274    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:37.284442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:37.284381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:37.386682 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:37.386649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:37.522542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:37.522514    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:37.836263 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:37.836209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:38.327705 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:38.327672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:40.420805 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.420770    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"]
Sep 10 23:53:40.425468 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.425447    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.428992 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.428967    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b88386\"/\"explicit-model-b88386-kserve-self-signed-certs\""
Sep 10 23:53:40.428992 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.428988    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b88386\"/\"default-dockercfg-rkkzx\""
Sep 10 23:53:40.429153 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.428997    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b88386\"/\"openshift-service-ca.crt\""
Sep 10 23:53:40.429153 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.429038    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b88386\"/\"kube-root-ca.crt\""
Sep 10 23:53:40.435764 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.435740    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"]
Sep 10 23:53:40.528946 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.528895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:40.553049 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.553026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-dshm\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.553162 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.553076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-home\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.553162 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.553108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-model-cache\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.553247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.553155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-tmp-dir\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.553247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.553211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br46h\" (UniqueName: \"kubernetes.io/projected/0bb7b79d-1722-4137-a160-9340a92d5186-kube-api-access-br46h\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.553247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.553243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb7b79d-1722-4137-a160-9340a92d5186-tls-certs\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654125 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-model-cache\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654259 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-tmp-dir\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654259 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br46h\" (UniqueName: \"kubernetes.io/projected/0bb7b79d-1722-4137-a160-9340a92d5186-kube-api-access-br46h\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654259 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb7b79d-1722-4137-a160-9340a92d5186-tls-certs\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654390 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-dshm\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654390 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-home\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654565 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-tmp-dir\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654600    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-model-cache\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.654669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.654625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-home\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.656615 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.656596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-dshm\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.656795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.656775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb7b79d-1722-4137-a160-9340a92d5186-tls-certs\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.662147 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.662126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br46h\" (UniqueName: \"kubernetes.io/projected/0bb7b79d-1722-4137-a160-9340a92d5186-kube-api-access-br46h\") pod \"explicit-model-b88386-kserve-7bd5d9f675-wbc5k\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") " pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:40.735140 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:40.735116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:41.007201 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:41.007172    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"]
Sep 10 23:53:41.009065 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:53:41.009038    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bb7b79d_1722_4137_a160_9340a92d5186.slice/crio-ebaa17af17354020ac14caf2880343c0d5581687c0a71b6b091055d2ef42c88c WatchSource:0}: Error finding container ebaa17af17354020ac14caf2880343c0d5581687c0a71b6b091055d2ef42c88c: Status 404 returned error can't find the container with id ebaa17af17354020ac14caf2880343c0d5581687c0a71b6b091055d2ef42c88c
Sep 10 23:53:41.146547 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:41.146515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:41.304117 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:41.304047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:41.447449 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:41.447412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k" event={"ID":"0bb7b79d-1722-4137-a160-9340a92d5186","Type":"ContainerStarted","Data":"bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320"}
Sep 10 23:53:41.447449 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:41.447453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k" event={"ID":"0bb7b79d-1722-4137-a160-9340a92d5186","Type":"ContainerStarted","Data":"ebaa17af17354020ac14caf2880343c0d5581687c0a71b6b091055d2ef42c88c"}
Sep 10 23:53:41.471116 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:41.471062    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k" podStartSLOduration=1.471048631 podStartE2EDuration="1.471048631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:41.470094652 +0000 UTC m=+1033.301122209" watchObservedRunningTime="2026-09-10 23:53:41.471048631 +0000 UTC m=+1033.302076163"
Sep 10 23:53:42.288698 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.288668    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:42.391782 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.391749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:42.623580 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.623497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:42.764004 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.763970    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:42.858301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.858268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:42.862749 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.862723    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"]
Sep 10 23:53:42.866183 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.866168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:42.868628 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.868602    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-24749012\""
Sep 10 23:53:42.868744 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.868727    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-24749012-dockercfg-28cfb\""
Sep 10 23:53:42.878973 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.878890    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"]
Sep 10 23:53:42.909193 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.909171    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"]
Sep 10 23:53:42.912414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.912399    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:42.928312 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.928280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"]
Sep 10 23:53:42.973365 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.973343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8r9\" (UniqueName: \"kubernetes.io/projected/96197fa8-b94b-4bb6-8d62-0a93b9202171-kube-api-access-7l8r9\") pod \"payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8\" (UID: \"96197fa8-b94b-4bb6-8d62-0a93b9202171\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:42.973471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.973376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnbbp\" (UniqueName: \"kubernetes.io/projected/d5e859d3-d359-4806-a8dd-f495bece995d-kube-api-access-nnbbp\") pod \"payload-processing-e2e-mt-24749012-56fd74874f-b2zjz\" (UID: \"d5e859d3-d359-4806-a8dd-f495bece995d\") " pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:42.973471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.973456    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e859d3-d359-4806-a8dd-f495bece995d-plugins-config-volume\") pod \"payload-processing-e2e-mt-24749012-56fd74874f-b2zjz\" (UID: \"d5e859d3-d359-4806-a8dd-f495bece995d\") " pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:42.973564 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:42.973498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96197fa8-b94b-4bb6-8d62-0a93b9202171-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8\" (UID: \"96197fa8-b94b-4bb6-8d62-0a93b9202171\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:43.074528 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.074501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e859d3-d359-4806-a8dd-f495bece995d-plugins-config-volume\") pod \"payload-processing-e2e-mt-24749012-56fd74874f-b2zjz\" (UID: \"d5e859d3-d359-4806-a8dd-f495bece995d\") " pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:43.074651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.074565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96197fa8-b94b-4bb6-8d62-0a93b9202171-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8\" (UID: \"96197fa8-b94b-4bb6-8d62-0a93b9202171\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:43.074651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.074606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8r9\" (UniqueName: \"kubernetes.io/projected/96197fa8-b94b-4bb6-8d62-0a93b9202171-kube-api-access-7l8r9\") pod \"payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8\" (UID: \"96197fa8-b94b-4bb6-8d62-0a93b9202171\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:43.074651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.074633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnbbp\" (UniqueName: \"kubernetes.io/projected/d5e859d3-d359-4806-a8dd-f495bece995d-kube-api-access-nnbbp\") pod \"payload-processing-e2e-mt-24749012-56fd74874f-b2zjz\" (UID: \"d5e859d3-d359-4806-a8dd-f495bece995d\") " pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:43.075203 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.075186    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e859d3-d359-4806-a8dd-f495bece995d-plugins-config-volume\") pod \"payload-processing-e2e-mt-24749012-56fd74874f-b2zjz\" (UID: \"d5e859d3-d359-4806-a8dd-f495bece995d\") " pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:43.075374 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.075357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96197fa8-b94b-4bb6-8d62-0a93b9202171-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8\" (UID: \"96197fa8-b94b-4bb6-8d62-0a93b9202171\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:43.096602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.096581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8r9\" (UniqueName: \"kubernetes.io/projected/96197fa8-b94b-4bb6-8d62-0a93b9202171-kube-api-access-7l8r9\") pod \"payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8\" (UID: \"96197fa8-b94b-4bb6-8d62-0a93b9202171\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:43.098469 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.098446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnbbp\" (UniqueName: \"kubernetes.io/projected/d5e859d3-d359-4806-a8dd-f495bece995d-kube-api-access-nnbbp\") pod \"payload-processing-e2e-mt-24749012-56fd74874f-b2zjz\" (UID: \"d5e859d3-d359-4806-a8dd-f495bece995d\") " pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:43.176372 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.176300    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:43.222427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.222390    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:43.452389 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.452365    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"]
Sep 10 23:53:43.454703 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:53:43.454674    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96197fa8_b94b_4bb6_8d62_0a93b9202171.slice/crio-c4274e64f0438a576fb2de06e64b132958896782802f76a1ac0780df54c94892 WatchSource:0}: Error finding container c4274e64f0438a576fb2de06e64b132958896782802f76a1ac0780df54c94892: Status 404 returned error can't find the container with id c4274e64f0438a576fb2de06e64b132958896782802f76a1ac0780df54c94892
Sep 10 23:53:43.461467 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.461443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8" event={"ID":"96197fa8-b94b-4bb6-8d62-0a93b9202171","Type":"ContainerStarted","Data":"c4274e64f0438a576fb2de06e64b132958896782802f76a1ac0780df54c94892"}
Sep 10 23:53:43.508903 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:43.508882    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"]
Sep 10 23:53:43.511884 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:53:43.511862    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e859d3_d359_4806_a8dd_f495bece995d.slice/crio-31bec125b4bb7a95d362f9a80b3a77a5a13f7438042c821dccdebeb7260dc4f4 WatchSource:0}: Error finding container 31bec125b4bb7a95d362f9a80b3a77a5a13f7438042c821dccdebeb7260dc4f4: Status 404 returned error can't find the container with id 31bec125b4bb7a95d362f9a80b3a77a5a13f7438042c821dccdebeb7260dc4f4
Sep 10 23:53:44.468733 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:44.468687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8" event={"ID":"96197fa8-b94b-4bb6-8d62-0a93b9202171","Type":"ContainerStarted","Data":"4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5"}
Sep 10 23:53:44.469227 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:44.468762    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:44.470415 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:44.470358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz" event={"ID":"d5e859d3-d359-4806-a8dd-f495bece995d","Type":"ContainerStarted","Data":"8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da"}
Sep 10 23:53:44.470523 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:44.470436    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:44.470523 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:44.470454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz" event={"ID":"d5e859d3-d359-4806-a8dd-f495bece995d","Type":"ContainerStarted","Data":"31bec125b4bb7a95d362f9a80b3a77a5a13f7438042c821dccdebeb7260dc4f4"}
Sep 10 23:53:44.486641 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:44.486587    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8" podStartSLOduration=2.486569804 podStartE2EDuration="2.486569804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:44.483974388 +0000 UTC m=+1036.315001936" watchObservedRunningTime="2026-09-10 23:53:44.486569804 +0000 UTC m=+1036.317597341"
Sep 10 23:53:44.499945 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:44.499889    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz" podStartSLOduration=2.499876262 podStartE2EDuration="2.499876262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:44.499692107 +0000 UTC m=+1036.330719645" watchObservedRunningTime="2026-09-10 23:53:44.499876262 +0000 UTC m=+1036.330903821"
Sep 10 23:53:48.015649 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:48.015613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:48.115239 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:48.115202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:48.913522 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:48.913489    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"]
Sep 10 23:53:48.948989 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:48.948958    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:49.248369 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:49.248341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:50.735596 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:50.735560    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:53:55.478994 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:55.478963    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:53:55.479383 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:55.479065    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:53:59.502063 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.502035    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"]
Sep 10 23:53:59.528786 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.528752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:59.548591 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.548551    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn" podUID="00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" containerName="maas-api" containerID="cri-o://ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8" gracePeriod=30
Sep 10 23:53:59.603447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.603417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:59.669152 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.669090    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:59.770669 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.770597    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:53:59.805608 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.805585    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:53:59.919264 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.919235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-metrics-certs\") pod \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") "
Sep 10 23:53:59.919442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.919275    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-openshift-service-ca\") pod \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") "
Sep 10 23:53:59.919442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.919377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-maas-api-tls\") pod \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") "
Sep 10 23:53:59.919442 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.919429    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwt9k\" (UniqueName: \"kubernetes.io/projected/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-kube-api-access-nwt9k\") pod \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\" (UID: \"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5\") "
Sep 10 23:53:59.919652 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.919622    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" (UID: "00a868a9-6c4f-4a39-a7db-fe3cf39d10a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:53:59.919779 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.919711    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:53:59.921204 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.921182    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" (UID: "00a868a9-6c4f-4a39-a7db-fe3cf39d10a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:53:59.921320 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.921294    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-kube-api-access-nwt9k" (OuterVolumeSpecName: "kube-api-access-nwt9k") pod "00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" (UID: "00a868a9-6c4f-4a39-a7db-fe3cf39d10a5"). InnerVolumeSpecName "kube-api-access-nwt9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:53:59.921385 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:53:59.921339    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" (UID: "00a868a9-6c4f-4a39-a7db-fe3cf39d10a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:00.020735 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.020664    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:00.020735 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.020692    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwt9k\" (UniqueName: \"kubernetes.io/projected/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-kube-api-access-nwt9k\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:00.020735 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.020705    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:00.059449 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.059419    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:00.560476 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.560440    2562 generic.go:356] "Generic (PLEG): container finished" podID="00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" containerID="ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8" exitCode=0
Sep 10 23:54:00.560872 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.560499    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"
Sep 10 23:54:00.560872 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.560525    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn" event={"ID":"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5","Type":"ContainerDied","Data":"ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8"}
Sep 10 23:54:00.560872 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.560609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn" event={"ID":"00a868a9-6c4f-4a39-a7db-fe3cf39d10a5","Type":"ContainerDied","Data":"bbbea8006a74eeadad1aafb37dda7a85e1e9b34f6dd69f31f783f1b4513a11e6"}
Sep 10 23:54:00.560872 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.560637    2562 scope.go:117] "RemoveContainer" containerID="ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8"
Sep 10 23:54:00.573222 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.573202    2562 scope.go:117] "RemoveContainer" containerID="ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8"
Sep 10 23:54:00.573482 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:00.573461    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8\": container with ID starting with ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8 not found: ID does not exist" containerID="ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8"
Sep 10 23:54:00.573565 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.573490    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8"} err="failed to get container status \"ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8\": rpc error: code = NotFound desc = could not find container \"ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8\": container with ID starting with ff9f39662dd684b3503393d1331c84c2e4d90070e2007bd2a8ccb3212bd2eea8 not found: ID does not exist"
Sep 10 23:54:00.584142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.584123    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"]
Sep 10 23:54:00.588153 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.588132    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b88386-f9fd4d766-67scn"]
Sep 10 23:54:00.655895 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:00.655861    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" path="/var/lib/kubelet/pods/00a868a9-6c4f-4a39-a7db-fe3cf39d10a5/volumes"
Sep 10 23:54:03.232097 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:03.232059    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:03.933627 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:03.933588    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k" podUID="0bb7b79d-1722-4137-a160-9340a92d5186" containerName="main" containerID="cri-o://bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320" gracePeriod=45
Sep 10 23:54:04.190601 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.190551    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:54:04.357157 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357115    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-model-cache\") pod \"0bb7b79d-1722-4137-a160-9340a92d5186\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") "
Sep 10 23:54:04.357577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-dshm\") pod \"0bb7b79d-1722-4137-a160-9340a92d5186\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") "
Sep 10 23:54:04.357577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357254    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-tmp-dir\") pod \"0bb7b79d-1722-4137-a160-9340a92d5186\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") "
Sep 10 23:54:04.357577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357273    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb7b79d-1722-4137-a160-9340a92d5186-tls-certs\") pod \"0bb7b79d-1722-4137-a160-9340a92d5186\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") "
Sep 10 23:54:04.357577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357292    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br46h\" (UniqueName: \"kubernetes.io/projected/0bb7b79d-1722-4137-a160-9340a92d5186-kube-api-access-br46h\") pod \"0bb7b79d-1722-4137-a160-9340a92d5186\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") "
Sep 10 23:54:04.357577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357292    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bb7b79d-1722-4137-a160-9340a92d5186" (UID: "0bb7b79d-1722-4137-a160-9340a92d5186"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:04.357577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-home\") pod \"0bb7b79d-1722-4137-a160-9340a92d5186\" (UID: \"0bb7b79d-1722-4137-a160-9340a92d5186\") "
Sep 10 23:54:04.357577 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357432    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bb7b79d-1722-4137-a160-9340a92d5186" (UID: "0bb7b79d-1722-4137-a160-9340a92d5186"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:04.357974 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357639    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-home" (OuterVolumeSpecName: "home") pod "0bb7b79d-1722-4137-a160-9340a92d5186" (UID: "0bb7b79d-1722-4137-a160-9340a92d5186"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:04.357974 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357656    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:04.357974 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.357678    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:04.359374 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.359351    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-dshm" (OuterVolumeSpecName: "dshm") pod "0bb7b79d-1722-4137-a160-9340a92d5186" (UID: "0bb7b79d-1722-4137-a160-9340a92d5186"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:04.359476 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.359411    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb7b79d-1722-4137-a160-9340a92d5186-kube-api-access-br46h" (OuterVolumeSpecName: "kube-api-access-br46h") pod "0bb7b79d-1722-4137-a160-9340a92d5186" (UID: "0bb7b79d-1722-4137-a160-9340a92d5186"). InnerVolumeSpecName "kube-api-access-br46h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:04.359476 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.359420    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb7b79d-1722-4137-a160-9340a92d5186-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bb7b79d-1722-4137-a160-9340a92d5186" (UID: "0bb7b79d-1722-4137-a160-9340a92d5186"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:04.458680 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.458616    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:04.458680 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.458639    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb7b79d-1722-4137-a160-9340a92d5186-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:04.458680 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.458650    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb7b79d-1722-4137-a160-9340a92d5186-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:04.458680 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.458660    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br46h\" (UniqueName: \"kubernetes.io/projected/0bb7b79d-1722-4137-a160-9340a92d5186-kube-api-access-br46h\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:04.582504 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.582469    2562 generic.go:356] "Generic (PLEG): container finished" podID="0bb7b79d-1722-4137-a160-9340a92d5186" containerID="bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320" exitCode=0
Sep 10 23:54:04.582647 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.582518    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"
Sep 10 23:54:04.582647 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.582555    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k" event={"ID":"0bb7b79d-1722-4137-a160-9340a92d5186","Type":"ContainerDied","Data":"bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320"}
Sep 10 23:54:04.582647 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.582613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k" event={"ID":"0bb7b79d-1722-4137-a160-9340a92d5186","Type":"ContainerDied","Data":"ebaa17af17354020ac14caf2880343c0d5581687c0a71b6b091055d2ef42c88c"}
Sep 10 23:54:04.582647 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.582635    2562 scope.go:117] "RemoveContainer" containerID="bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320"
Sep 10 23:54:04.591996 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.591976    2562 scope.go:117] "RemoveContainer" containerID="bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320"
Sep 10 23:54:04.592268 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:04.592251    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320\": container with ID starting with bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320 not found: ID does not exist" containerID="bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320"
Sep 10 23:54:04.592310 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.592277    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320"} err="failed to get container status \"bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320\": rpc error: code = NotFound desc = could not find container \"bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320\": container with ID starting with bc7d4b145c366c454451e7214aadb1f1319b101f79838246ef5918c9f4b42320 not found: ID does not exist"
Sep 10 23:54:04.604992 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.604973    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"]
Sep 10 23:54:04.611053 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.611033    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b88386/explicit-model-b88386-kserve-7bd5d9f675-wbc5k"]
Sep 10 23:54:04.655548 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:04.655524    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb7b79d-1722-4137-a160-9340a92d5186" path="/var/lib/kubelet/pods/0bb7b79d-1722-4137-a160-9340a92d5186/volumes"
Sep 10 23:54:05.993954 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:05.993899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:06.397336 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:06.397263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:06.498774 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:06.498738    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:06.659646 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:06.659563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:06.754913 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:06.754883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:07.031510 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:07.031479    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:07.131815 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:07.131786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:07.477698 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:07.477660    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:07.712393 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:07.712360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:08.550506 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:08.550475    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:08.643423 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:08.643354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:08.742537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:08.742507    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:12.997344 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:12.997290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:13.066548 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:13.066503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:23.971088 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:23.971010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:24.012425 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.012393    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"]
Sep 10 23:54:24.012674 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.012642    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz" podUID="d5e859d3-d359-4806-a8dd-f495bece995d" containerName="payload-processing" containerID="cri-o://8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da" gracePeriod=30
Sep 10 23:54:24.018976 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.018952    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"]
Sep 10 23:54:24.019604 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.019186    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8" podUID="96197fa8-b94b-4bb6-8d62-0a93b9202171" containerName="payload-pre-processing" containerID="cri-o://4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5" gracePeriod=30
Sep 10 23:54:24.058191 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.058165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:24.128183 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.128156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:24.197282 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.197210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:24.276528 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.276508    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:54:24.310792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.310768    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:54:24.317116 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.317094    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e859d3-d359-4806-a8dd-f495bece995d-plugins-config-volume\") pod \"d5e859d3-d359-4806-a8dd-f495bece995d\" (UID: \"d5e859d3-d359-4806-a8dd-f495bece995d\") "
Sep 10 23:54:24.317225 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.317166    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnbbp\" (UniqueName: \"kubernetes.io/projected/d5e859d3-d359-4806-a8dd-f495bece995d-kube-api-access-nnbbp\") pod \"d5e859d3-d359-4806-a8dd-f495bece995d\" (UID: \"d5e859d3-d359-4806-a8dd-f495bece995d\") "
Sep 10 23:54:24.317517 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.317483    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e859d3-d359-4806-a8dd-f495bece995d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5e859d3-d359-4806-a8dd-f495bece995d" (UID: "d5e859d3-d359-4806-a8dd-f495bece995d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:24.319263 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.319243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e859d3-d359-4806-a8dd-f495bece995d-kube-api-access-nnbbp" (OuterVolumeSpecName: "kube-api-access-nnbbp") pod "d5e859d3-d359-4806-a8dd-f495bece995d" (UID: "d5e859d3-d359-4806-a8dd-f495bece995d"). InnerVolumeSpecName "kube-api-access-nnbbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:24.417639 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.417611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96197fa8-b94b-4bb6-8d62-0a93b9202171-plugins-config-volume\") pod \"96197fa8-b94b-4bb6-8d62-0a93b9202171\" (UID: \"96197fa8-b94b-4bb6-8d62-0a93b9202171\") "
Sep 10 23:54:24.417784 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.417652    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l8r9\" (UniqueName: \"kubernetes.io/projected/96197fa8-b94b-4bb6-8d62-0a93b9202171-kube-api-access-7l8r9\") pod \"96197fa8-b94b-4bb6-8d62-0a93b9202171\" (UID: \"96197fa8-b94b-4bb6-8d62-0a93b9202171\") "
Sep 10 23:54:24.417882 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.417864    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnbbp\" (UniqueName: \"kubernetes.io/projected/d5e859d3-d359-4806-a8dd-f495bece995d-kube-api-access-nnbbp\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:24.417951 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.417889    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e859d3-d359-4806-a8dd-f495bece995d-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:24.418044 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.418026    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96197fa8-b94b-4bb6-8d62-0a93b9202171-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96197fa8-b94b-4bb6-8d62-0a93b9202171" (UID: "96197fa8-b94b-4bb6-8d62-0a93b9202171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:24.419630 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.419606    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96197fa8-b94b-4bb6-8d62-0a93b9202171-kube-api-access-7l8r9" (OuterVolumeSpecName: "kube-api-access-7l8r9") pod "96197fa8-b94b-4bb6-8d62-0a93b9202171" (UID: "96197fa8-b94b-4bb6-8d62-0a93b9202171"). InnerVolumeSpecName "kube-api-access-7l8r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:24.519292 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.519224    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96197fa8-b94b-4bb6-8d62-0a93b9202171-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:24.519292 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.519249    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8r9\" (UniqueName: \"kubernetes.io/projected/96197fa8-b94b-4bb6-8d62-0a93b9202171-kube-api-access-7l8r9\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:24.589054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.589025    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:24.680679 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.680640    2562 generic.go:356] "Generic (PLEG): container finished" podID="d5e859d3-d359-4806-a8dd-f495bece995d" containerID="8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da" exitCode=0
Sep 10 23:54:24.680777 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.680696    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"
Sep 10 23:54:24.680777 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.680718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz" event={"ID":"d5e859d3-d359-4806-a8dd-f495bece995d","Type":"ContainerDied","Data":"8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da"}
Sep 10 23:54:24.680919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.680777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz" event={"ID":"d5e859d3-d359-4806-a8dd-f495bece995d","Type":"ContainerDied","Data":"31bec125b4bb7a95d362f9a80b3a77a5a13f7438042c821dccdebeb7260dc4f4"}
Sep 10 23:54:24.680919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.680813    2562 scope.go:117] "RemoveContainer" containerID="8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da"
Sep 10 23:54:24.682199 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.682152    2562 generic.go:356] "Generic (PLEG): container finished" podID="96197fa8-b94b-4bb6-8d62-0a93b9202171" containerID="4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5" exitCode=0
Sep 10 23:54:24.682285 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.682205    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"
Sep 10 23:54:24.682285 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.682238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8" event={"ID":"96197fa8-b94b-4bb6-8d62-0a93b9202171","Type":"ContainerDied","Data":"4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5"}
Sep 10 23:54:24.682285 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.682271    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8" event={"ID":"96197fa8-b94b-4bb6-8d62-0a93b9202171","Type":"ContainerDied","Data":"c4274e64f0438a576fb2de06e64b132958896782802f76a1ac0780df54c94892"}
Sep 10 23:54:24.690566 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.690552    2562 scope.go:117] "RemoveContainer" containerID="8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da"
Sep 10 23:54:24.690862 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:24.690834    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da\": container with ID starting with 8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da not found: ID does not exist" containerID="8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da"
Sep 10 23:54:24.690961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.690874    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da"} err="failed to get container status \"8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da\": rpc error: code = NotFound desc = could not find container \"8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da\": container with ID starting with 8eaeda1e9b116ae819dcfeb5a38de348b1b3914949a9b12c8bf19330103e33da not found: ID does not exist"
Sep 10 23:54:24.690961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.690895    2562 scope.go:117] "RemoveContainer" containerID="4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5"
Sep 10 23:54:24.698307 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.698290    2562 scope.go:117] "RemoveContainer" containerID="4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5"
Sep 10 23:54:24.698573 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:24.698555    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5\": container with ID starting with 4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5 not found: ID does not exist" containerID="4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5"
Sep 10 23:54:24.698619 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.698583    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5"} err="failed to get container status \"4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5\": rpc error: code = NotFound desc = could not find container \"4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5\": container with ID starting with 4f54678b631645b7625ae3eda54961a4af4f83f847ea05e9b90bbd28277902b5 not found: ID does not exist"
Sep 10 23:54:24.701918 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.701897    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"]
Sep 10 23:54:24.705960 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.705941    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-24749012-bf96ffd86-67fm8"]
Sep 10 23:54:24.717869 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.717849    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"]
Sep 10 23:54:24.721634 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:24.721616    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-24749012-56fd74874f-b2zjz"]
Sep 10 23:54:26.655752 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:26.655716    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96197fa8-b94b-4bb6-8d62-0a93b9202171" path="/var/lib/kubelet/pods/96197fa8-b94b-4bb6-8d62-0a93b9202171/volumes"
Sep 10 23:54:26.656149 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:26.656086    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e859d3-d359-4806-a8dd-f495bece995d" path="/var/lib/kubelet/pods/d5e859d3-d359-4806-a8dd-f495bece995d/volumes"
Sep 10 23:54:27.408992 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:27.408959    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"]
Sep 10 23:54:27.451629 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:27.451601    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:28.357236 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:28.357203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:28.473333 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:28.473302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:28.679536 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:28.679473    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:28.795896 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:28.795870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:29.116491 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:29.116456    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"]
Sep 10 23:54:29.141252 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:29.141227    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:29.376813 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:29.376728    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.203581 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:31.203545    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.305392 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:31.305360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.468041 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:31.467962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.778029 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:31.777943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:31.943166 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:31.943134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:32.097059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:32.096975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:33.836005 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:33.835976    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:39.249040 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:39.249007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:39.455095 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:39.455062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:39.620292 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:39.620212    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:39.715541 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:39.715514    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:39.971427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:39.971395    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"]
Sep 10 23:54:39.971827 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:39.971634    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw" podUID="8a698155-f53a-427c-989f-abdaab2357d5" containerName="maas-api" containerID="cri-o://a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0" gracePeriod=30
Sep 10 23:54:40.017605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.017577    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:40.048118 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.048089    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"]
Sep 10 23:54:40.048440 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.048405    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm" podUID="0f4bd004-efd1-471c-9e1f-74580dbe5a37" containerName="payload-processing" containerID="cri-o://af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659" gracePeriod=30
Sep 10 23:54:40.129714 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.129663    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:40.222189 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.222071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:40.285798 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.285770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:40.301800 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.301779    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:54:40.328102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.328070    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:54:40.341132 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.341108    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hmqb\" (UniqueName: \"kubernetes.io/projected/8a698155-f53a-427c-989f-abdaab2357d5-kube-api-access-9hmqb\") pod \"8a698155-f53a-427c-989f-abdaab2357d5\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") "
Sep 10 23:54:40.341258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.341222    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-metrics-certs\") pod \"8a698155-f53a-427c-989f-abdaab2357d5\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") "
Sep 10 23:54:40.341321 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.341256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-maas-api-tls\") pod \"8a698155-f53a-427c-989f-abdaab2357d5\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") "
Sep 10 23:54:40.341321 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.341289    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698155-f53a-427c-989f-abdaab2357d5-openshift-service-ca\") pod \"8a698155-f53a-427c-989f-abdaab2357d5\" (UID: \"8a698155-f53a-427c-989f-abdaab2357d5\") "
Sep 10 23:54:40.341668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.341643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a698155-f53a-427c-989f-abdaab2357d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a698155-f53a-427c-989f-abdaab2357d5" (UID: "8a698155-f53a-427c-989f-abdaab2357d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:40.343161 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.343140    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a698155-f53a-427c-989f-abdaab2357d5-kube-api-access-9hmqb" (OuterVolumeSpecName: "kube-api-access-9hmqb") pod "8a698155-f53a-427c-989f-abdaab2357d5" (UID: "8a698155-f53a-427c-989f-abdaab2357d5"). InnerVolumeSpecName "kube-api-access-9hmqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:40.343698 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.343677    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a698155-f53a-427c-989f-abdaab2357d5" (UID: "8a698155-f53a-427c-989f-abdaab2357d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:40.343763 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.343705    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a698155-f53a-427c-989f-abdaab2357d5" (UID: "8a698155-f53a-427c-989f-abdaab2357d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:40.442351 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.442327    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvc5b\" (UniqueName: \"kubernetes.io/projected/0f4bd004-efd1-471c-9e1f-74580dbe5a37-kube-api-access-wvc5b\") pod \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\" (UID: \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\") "
Sep 10 23:54:40.442515 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.442403    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4bd004-efd1-471c-9e1f-74580dbe5a37-plugins-config-volume\") pod \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\" (UID: \"0f4bd004-efd1-471c-9e1f-74580dbe5a37\") "
Sep 10 23:54:40.442657 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.442641    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:40.442718 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.442663    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a698155-f53a-427c-989f-abdaab2357d5-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:40.442718 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.442676    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698155-f53a-427c-989f-abdaab2357d5-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:40.442718 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.442690    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hmqb\" (UniqueName: \"kubernetes.io/projected/8a698155-f53a-427c-989f-abdaab2357d5-kube-api-access-9hmqb\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:40.442816 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.442722    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4bd004-efd1-471c-9e1f-74580dbe5a37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f4bd004-efd1-471c-9e1f-74580dbe5a37" (UID: "0f4bd004-efd1-471c-9e1f-74580dbe5a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:40.444261 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.444241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4bd004-efd1-471c-9e1f-74580dbe5a37-kube-api-access-wvc5b" (OuterVolumeSpecName: "kube-api-access-wvc5b") pod "0f4bd004-efd1-471c-9e1f-74580dbe5a37" (UID: "0f4bd004-efd1-471c-9e1f-74580dbe5a37"). InnerVolumeSpecName "kube-api-access-wvc5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:40.543557 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.543499    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvc5b\" (UniqueName: \"kubernetes.io/projected/0f4bd004-efd1-471c-9e1f-74580dbe5a37-kube-api-access-wvc5b\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:40.543557 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.543520    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4bd004-efd1-471c-9e1f-74580dbe5a37-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:40.583540 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.583516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:40.759363 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.759335    2562 generic.go:356] "Generic (PLEG): container finished" podID="0f4bd004-efd1-471c-9e1f-74580dbe5a37" containerID="af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659" exitCode=0
Sep 10 23:54:40.759493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.759374    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"
Sep 10 23:54:40.759493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.759429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm" event={"ID":"0f4bd004-efd1-471c-9e1f-74580dbe5a37","Type":"ContainerDied","Data":"af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659"}
Sep 10 23:54:40.759493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.759482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm" event={"ID":"0f4bd004-efd1-471c-9e1f-74580dbe5a37","Type":"ContainerDied","Data":"371a46a682ccb6e175ea602d2205380fa86303f1e646050531e85d04500a3d1f"}
Sep 10 23:54:40.759618 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.759514    2562 scope.go:117] "RemoveContainer" containerID="af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659"
Sep 10 23:54:40.761213 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.761079    2562 generic.go:356] "Generic (PLEG): container finished" podID="8a698155-f53a-427c-989f-abdaab2357d5" containerID="a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0" exitCode=0
Sep 10 23:54:40.761213 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.761148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw" event={"ID":"8a698155-f53a-427c-989f-abdaab2357d5","Type":"ContainerDied","Data":"a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0"}
Sep 10 23:54:40.761213 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.761178    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"
Sep 10 23:54:40.761213 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.761188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw" event={"ID":"8a698155-f53a-427c-989f-abdaab2357d5","Type":"ContainerDied","Data":"b24122c06406f0bb8ccfc79fc688fbc7e3e2c38cfb25c3ad94335c6c19f26aa5"}
Sep 10 23:54:40.768091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.768069    2562 scope.go:117] "RemoveContainer" containerID="af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659"
Sep 10 23:54:40.768351 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:40.768332    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659\": container with ID starting with af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659 not found: ID does not exist" containerID="af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659"
Sep 10 23:54:40.768445 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.768367    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659"} err="failed to get container status \"af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659\": rpc error: code = NotFound desc = could not find container \"af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659\": container with ID starting with af11e1af818c347aae3567b63c400b622395a535d49e2a9fbf702fe75c973659 not found: ID does not exist"
Sep 10 23:54:40.768445 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.768385    2562 scope.go:117] "RemoveContainer" containerID="a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0"
Sep 10 23:54:40.775717 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.775701    2562 scope.go:117] "RemoveContainer" containerID="a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0"
Sep 10 23:54:40.775969 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:40.775951    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0\": container with ID starting with a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0 not found: ID does not exist" containerID="a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0"
Sep 10 23:54:40.776036 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.775974    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0"} err="failed to get container status \"a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0\": rpc error: code = NotFound desc = could not find container \"a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0\": container with ID starting with a1ca704037ac43ab9f9d89dd4d1eb50c43d4172d38d8e4ae4957cfc008d258e0 not found: ID does not exist"
Sep 10 23:54:40.783080 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.783056    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"]
Sep 10 23:54:40.800058 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.799998    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fe290-78558c7d55-hjxdm"]
Sep 10 23:54:40.825348 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.825324    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"]
Sep 10 23:54:40.832427 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:40.832409    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fe290-5589cfdd5f-lwnbw"]
Sep 10 23:54:42.427688 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.427650    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn" podUID="2da23751-134e-4c4d-9527-fe292c212d2e" containerName="main" containerID="cri-o://07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61" gracePeriod=45
Sep 10 23:54:42.656301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.656272    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f4bd004-efd1-471c-9e1f-74580dbe5a37" path="/var/lib/kubelet/pods/0f4bd004-efd1-471c-9e1f-74580dbe5a37/volumes"
Sep 10 23:54:42.656661 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.656648    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a698155-f53a-427c-989f-abdaab2357d5" path="/var/lib/kubelet/pods/8a698155-f53a-427c-989f-abdaab2357d5/volumes"
Sep 10 23:54:42.683408 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.683390    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:54:42.761122 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761098    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da23751-134e-4c4d-9527-fe292c212d2e-tls-certs\") pod \"2da23751-134e-4c4d-9527-fe292c212d2e\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") "
Sep 10 23:54:42.761242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761134    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-tmp-dir\") pod \"2da23751-134e-4c4d-9527-fe292c212d2e\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") "
Sep 10 23:54:42.761242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761170    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-home\") pod \"2da23751-134e-4c4d-9527-fe292c212d2e\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") "
Sep 10 23:54:42.761242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761186    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-model-cache\") pod \"2da23751-134e-4c4d-9527-fe292c212d2e\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") "
Sep 10 23:54:42.761242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761236    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wplqc\" (UniqueName: \"kubernetes.io/projected/2da23751-134e-4c4d-9527-fe292c212d2e-kube-api-access-wplqc\") pod \"2da23751-134e-4c4d-9527-fe292c212d2e\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") "
Sep 10 23:54:42.761456 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761284    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-dshm\") pod \"2da23751-134e-4c4d-9527-fe292c212d2e\" (UID: \"2da23751-134e-4c4d-9527-fe292c212d2e\") "
Sep 10 23:54:42.761456 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2da23751-134e-4c4d-9527-fe292c212d2e" (UID: "2da23751-134e-4c4d-9527-fe292c212d2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:42.761456 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761401    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-home" (OuterVolumeSpecName: "home") pod "2da23751-134e-4c4d-9527-fe292c212d2e" (UID: "2da23751-134e-4c4d-9527-fe292c212d2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:42.761595 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761481    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2da23751-134e-4c4d-9527-fe292c212d2e" (UID: "2da23751-134e-4c4d-9527-fe292c212d2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:42.761656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761597    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:42.761656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761615    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:42.761656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.761631    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:42.763417 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.763393    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da23751-134e-4c4d-9527-fe292c212d2e-kube-api-access-wplqc" (OuterVolumeSpecName: "kube-api-access-wplqc") pod "2da23751-134e-4c4d-9527-fe292c212d2e" (UID: "2da23751-134e-4c4d-9527-fe292c212d2e"). InnerVolumeSpecName "kube-api-access-wplqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:42.763638 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.763614    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-dshm" (OuterVolumeSpecName: "dshm") pod "2da23751-134e-4c4d-9527-fe292c212d2e" (UID: "2da23751-134e-4c4d-9527-fe292c212d2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:42.763638 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.763627    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da23751-134e-4c4d-9527-fe292c212d2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2da23751-134e-4c4d-9527-fe292c212d2e" (UID: "2da23751-134e-4c4d-9527-fe292c212d2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:42.773251 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.773232    2562 generic.go:356] "Generic (PLEG): container finished" podID="2da23751-134e-4c4d-9527-fe292c212d2e" containerID="07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61" exitCode=0
Sep 10 23:54:42.773345 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.773293    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"
Sep 10 23:54:42.773345 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.773321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn" event={"ID":"2da23751-134e-4c4d-9527-fe292c212d2e","Type":"ContainerDied","Data":"07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61"}
Sep 10 23:54:42.773422 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.773363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn" event={"ID":"2da23751-134e-4c4d-9527-fe292c212d2e","Type":"ContainerDied","Data":"d64a5fd23582db72dc4272a70684dd83f1d5c3c681743bd8ea5ddfa22259383f"}
Sep 10 23:54:42.773422 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.773386    2562 scope.go:117] "RemoveContainer" containerID="07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61"
Sep 10 23:54:42.782058 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.781892    2562 scope.go:117] "RemoveContainer" containerID="07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61"
Sep 10 23:54:42.782213 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:42.782186    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61\": container with ID starting with 07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61 not found: ID does not exist" containerID="07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61"
Sep 10 23:54:42.782306 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.782225    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61"} err="failed to get container status \"07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61\": rpc error: code = NotFound desc = could not find container \"07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61\": container with ID starting with 07f7db4eb94fee0d92a97938849275c186206233f97a0de9b58b6b19eb571b61 not found: ID does not exist"
Sep 10 23:54:42.795078 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.795059    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"]
Sep 10 23:54:42.798654 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.798635    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1fe290/test-model-1fe290-kserve-695c9666cf-8mpzn"]
Sep 10 23:54:42.862551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.862530    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplqc\" (UniqueName: \"kubernetes.io/projected/2da23751-134e-4c4d-9527-fe292c212d2e-kube-api-access-wplqc\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:42.862551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.862550    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da23751-134e-4c4d-9527-fe292c212d2e-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:42.862668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:42.862560    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da23751-134e-4c4d-9527-fe292c212d2e-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:44.135489 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.135432    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9" podUID="f5925694-c4df-4f80-a1fb-c9a44472fc7f" containerName="main" containerID="cri-o://426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea" gracePeriod=45
Sep 10 23:54:44.377392 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.377364    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:54:44.474419 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474395    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tls-certs\") pod \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") "
Sep 10 23:54:44.474584 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-home\") pod \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") "
Sep 10 23:54:44.474584 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474498    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-dshm\") pod \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") "
Sep 10 23:54:44.474584 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474535    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gknzl\" (UniqueName: \"kubernetes.io/projected/f5925694-c4df-4f80-a1fb-c9a44472fc7f-kube-api-access-gknzl\") pod \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") "
Sep 10 23:54:44.474584 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tmp-dir\") pod \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") "
Sep 10 23:54:44.474807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474614    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-model-cache\") pod \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\" (UID: \"f5925694-c4df-4f80-a1fb-c9a44472fc7f\") "
Sep 10 23:54:44.474807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474777    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-home" (OuterVolumeSpecName: "home") pod "f5925694-c4df-4f80-a1fb-c9a44472fc7f" (UID: "f5925694-c4df-4f80-a1fb-c9a44472fc7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:44.474900 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474816    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5925694-c4df-4f80-a1fb-c9a44472fc7f" (UID: "f5925694-c4df-4f80-a1fb-c9a44472fc7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:44.474900 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.474869    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:44.475035 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.475014    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5925694-c4df-4f80-a1fb-c9a44472fc7f" (UID: "f5925694-c4df-4f80-a1fb-c9a44472fc7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:44.476629 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.476603    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-dshm" (OuterVolumeSpecName: "dshm") pod "f5925694-c4df-4f80-a1fb-c9a44472fc7f" (UID: "f5925694-c4df-4f80-a1fb-c9a44472fc7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:44.476719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.476682    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5925694-c4df-4f80-a1fb-c9a44472fc7f-kube-api-access-gknzl" (OuterVolumeSpecName: "kube-api-access-gknzl") pod "f5925694-c4df-4f80-a1fb-c9a44472fc7f" (UID: "f5925694-c4df-4f80-a1fb-c9a44472fc7f"). InnerVolumeSpecName "kube-api-access-gknzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:44.476719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.476682    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5925694-c4df-4f80-a1fb-c9a44472fc7f" (UID: "f5925694-c4df-4f80-a1fb-c9a44472fc7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:44.575768 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.575741    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:44.575768 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.575765    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gknzl\" (UniqueName: \"kubernetes.io/projected/f5925694-c4df-4f80-a1fb-c9a44472fc7f-kube-api-access-gknzl\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:44.575922 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.575777    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:44.575922 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.575786    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5925694-c4df-4f80-a1fb-c9a44472fc7f-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:44.575922 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.575794    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5925694-c4df-4f80-a1fb-c9a44472fc7f-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:54:44.656775 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.656749    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da23751-134e-4c4d-9527-fe292c212d2e" path="/var/lib/kubelet/pods/2da23751-134e-4c4d-9527-fe292c212d2e/volumes"
Sep 10 23:54:44.784380 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.784318    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5925694-c4df-4f80-a1fb-c9a44472fc7f" containerID="426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea" exitCode=0
Sep 10 23:54:44.784380 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.784355    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"
Sep 10 23:54:44.784551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.784406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9" event={"ID":"f5925694-c4df-4f80-a1fb-c9a44472fc7f","Type":"ContainerDied","Data":"426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea"}
Sep 10 23:54:44.784551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.784468    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9" event={"ID":"f5925694-c4df-4f80-a1fb-c9a44472fc7f","Type":"ContainerDied","Data":"7da9330a49f68c07edcd7a664fde67281286f36283317e7a50589c6661af6031"}
Sep 10 23:54:44.784551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.784489    2562 scope.go:117] "RemoveContainer" containerID="426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea"
Sep 10 23:54:44.792839 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.792824    2562 scope.go:117] "RemoveContainer" containerID="426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea"
Sep 10 23:54:44.793132 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:44.793114    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea\": container with ID starting with 426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea not found: ID does not exist" containerID="426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea"
Sep 10 23:54:44.793199 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.793137    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea"} err="failed to get container status \"426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea\": rpc error: code = NotFound desc = could not find container \"426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea\": container with ID starting with 426ded1b2ebbb46e9d3170965a49a1e49c5bd9de2598d38bd652f7d81879f9ea not found: ID does not exist"
Sep 10 23:54:44.801081 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.801061    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"]
Sep 10 23:54:44.805601 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:44.805575    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7f598e/test-model-7f598e-kserve-6d9b86c4cc-q7pp9"]
Sep 10 23:54:45.392467 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:45.392436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:45.494250 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:45.494221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:46.269810 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:46.269774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:46.590768 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:46.590690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:46.654772 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:46.654740    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5925694-c4df-4f80-a1fb-c9a44472fc7f" path="/var/lib/kubelet/pods/f5925694-c4df-4f80-a1fb-c9a44472fc7f/volumes"
Sep 10 23:54:46.941798 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:46.941769    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:47.073672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:47.073642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:47.314161 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:47.314088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:47.401210 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:47.401185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:47.850397 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:47.850363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:49.529694 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:49.529654    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:49.610823 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:49.610791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:50.729036 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729002    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"]
Sep 10 23:54:50.729414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729337    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb7b79d-1722-4137-a160-9340a92d5186" containerName="main"
Sep 10 23:54:50.729414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729349    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb7b79d-1722-4137-a160-9340a92d5186" containerName="main"
Sep 10 23:54:50.729414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729370    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96197fa8-b94b-4bb6-8d62-0a93b9202171" containerName="payload-pre-processing"
Sep 10 23:54:50.729414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729376    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="96197fa8-b94b-4bb6-8d62-0a93b9202171" containerName="payload-pre-processing"
Sep 10 23:54:50.729414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729387    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" containerName="maas-api"
Sep 10 23:54:50.729414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729394    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" containerName="maas-api"
Sep 10 23:54:50.729414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729405    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4bd004-efd1-471c-9e1f-74580dbe5a37" containerName="payload-processing"
Sep 10 23:54:50.729414 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729413    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4bd004-efd1-471c-9e1f-74580dbe5a37" containerName="payload-processing"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729422    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e859d3-d359-4806-a8dd-f495bece995d" containerName="payload-processing"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729427    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e859d3-d359-4806-a8dd-f495bece995d" containerName="payload-processing"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729434    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da23751-134e-4c4d-9527-fe292c212d2e" containerName="main"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729440    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da23751-134e-4c4d-9527-fe292c212d2e" containerName="main"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729447    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a698155-f53a-427c-989f-abdaab2357d5" containerName="maas-api"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729451    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a698155-f53a-427c-989f-abdaab2357d5" containerName="maas-api"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729456    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5925694-c4df-4f80-a1fb-c9a44472fc7f" containerName="main"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729461    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5925694-c4df-4f80-a1fb-c9a44472fc7f" containerName="main"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729512    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da23751-134e-4c4d-9527-fe292c212d2e" containerName="main"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729519    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e859d3-d359-4806-a8dd-f495bece995d" containerName="payload-processing"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729524    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a698155-f53a-427c-989f-abdaab2357d5" containerName="maas-api"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729529    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a868a9-6c4f-4a39-a7db-fe3cf39d10a5" containerName="maas-api"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729534    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="96197fa8-b94b-4bb6-8d62-0a93b9202171" containerName="payload-pre-processing"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729540    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5925694-c4df-4f80-a1fb-c9a44472fc7f" containerName="main"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729545    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4bd004-efd1-471c-9e1f-74580dbe5a37" containerName="payload-processing"
Sep 10 23:54:50.729668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.729549    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb7b79d-1722-4137-a160-9340a92d5186" containerName="main"
Sep 10 23:54:50.732771 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.732749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.735221 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.735201    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9tsnw\""
Sep 10 23:54:50.735389 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.735374    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b19d25-serving-cert\""
Sep 10 23:54:50.746641 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.746619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"]
Sep 10 23:54:50.825694 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.825661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.825867 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.825721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53923999-1831-452c-8399-e7e81365abc5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.825867 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.825832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgwm\" (UniqueName: \"kubernetes.io/projected/53923999-1831-452c-8399-e7e81365abc5-kube-api-access-dwgwm\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.826003 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.825896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.827689 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.827664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:50.926513 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.926489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgwm\" (UniqueName: \"kubernetes.io/projected/53923999-1831-452c-8399-e7e81365abc5-kube-api-access-dwgwm\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.926632 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.926522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.926632 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.926556    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.926632 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.926581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53923999-1831-452c-8399-e7e81365abc5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.926780 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:50.926677    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-serving-cert: secret "maas-api-e2e-shared-a-w2-b19d25-serving-cert" not found
Sep 10 23:54:50.926780 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:54:50.926758    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls podName:53923999-1831-452c-8399-e7e81365abc5 nodeName:}" failed. No retries permitted until 2026-09-10 23:54:51.426735951 +0000 UTC m=+1103.257763479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls") pod "maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh" (UID: "53923999-1831-452c-8399-e7e81365abc5") : secret "maas-api-e2e-shared-a-w2-b19d25-serving-cert" not found
Sep 10 23:54:50.927287 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.927267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53923999-1831-452c-8399-e7e81365abc5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.929017 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.928996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:50.930777 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.930754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:50.935820 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:50.935792    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgwm\" (UniqueName: \"kubernetes.io/projected/53923999-1831-452c-8399-e7e81365abc5-kube-api-access-dwgwm\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:51.096337 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.096257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:51.344204 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.344175    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:51.431434 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.431403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:51.432560 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.432532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:51.433239 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.433219    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"]
Sep 10 23:54:51.433991 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.433973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:51.436722 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.436707    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:51.439184 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.439162    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b19d25\""
Sep 10 23:54:51.439306 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.439260    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b19d25-dockercfg-mtbtx\""
Sep 10 23:54:51.452621 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.452600    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"]
Sep 10 23:54:51.516089 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.516064    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"]
Sep 10 23:54:51.519108 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.519086    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:51.532647 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.532627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j77d\" (UniqueName: \"kubernetes.io/projected/1aaf3e09-3091-4647-bb89-1011cf00d433-kube-api-access-2j77d\") pod \"payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576\" (UID: \"1aaf3e09-3091-4647-bb89-1011cf00d433\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:51.532744 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.532659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaf3e09-3091-4647-bb89-1011cf00d433-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576\" (UID: \"1aaf3e09-3091-4647-bb89-1011cf00d433\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:51.550258 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.550232    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"]
Sep 10 23:54:51.633436 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.633411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7x2\" (UniqueName: \"kubernetes.io/projected/8e1e418b-89a0-430d-b8fa-acece16ee0c4-kube-api-access-bb7x2\") pod \"payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp\" (UID: \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:51.633547 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.633451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j77d\" (UniqueName: \"kubernetes.io/projected/1aaf3e09-3091-4647-bb89-1011cf00d433-kube-api-access-2j77d\") pod \"payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576\" (UID: \"1aaf3e09-3091-4647-bb89-1011cf00d433\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:51.633547 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.633472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e418b-89a0-430d-b8fa-acece16ee0c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp\" (UID: \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:51.633547 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.633504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaf3e09-3091-4647-bb89-1011cf00d433-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576\" (UID: \"1aaf3e09-3091-4647-bb89-1011cf00d433\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:51.634198 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.634180    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaf3e09-3091-4647-bb89-1011cf00d433-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576\" (UID: \"1aaf3e09-3091-4647-bb89-1011cf00d433\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:51.644493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.644471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j77d\" (UniqueName: \"kubernetes.io/projected/1aaf3e09-3091-4647-bb89-1011cf00d433-kube-api-access-2j77d\") pod \"payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576\" (UID: \"1aaf3e09-3091-4647-bb89-1011cf00d433\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:51.645278 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.645258    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:51.734469 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.734439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7x2\" (UniqueName: \"kubernetes.io/projected/8e1e418b-89a0-430d-b8fa-acece16ee0c4-kube-api-access-bb7x2\") pod \"payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp\" (UID: \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:51.734785 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.734483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e418b-89a0-430d-b8fa-acece16ee0c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp\" (UID: \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:51.735173 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.735155    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e418b-89a0-430d-b8fa-acece16ee0c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp\" (UID: \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:51.752155 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.750388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7x2\" (UniqueName: \"kubernetes.io/projected/8e1e418b-89a0-430d-b8fa-acece16ee0c4-kube-api-access-bb7x2\") pod \"payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp\" (UID: \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:51.752155 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.751969    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:51.828563 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.828524    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:51.920907 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:51.920873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"]
Sep 10 23:54:51.921740 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:54:51.921713    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53923999_1831_452c_8399_e7e81365abc5.slice/crio-b0c48219ded1799edc2b9d81117f56404a60bb0495dabc6c9fb33eb759c194fb WatchSource:0}: Error finding container b0c48219ded1799edc2b9d81117f56404a60bb0495dabc6c9fb33eb759c194fb: Status 404 returned error can't find the container with id b0c48219ded1799edc2b9d81117f56404a60bb0495dabc6c9fb33eb759c194fb
Sep 10 23:54:52.022961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.022877    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"]
Sep 10 23:54:52.025868 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:54:52.025843    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaf3e09_3091_4647_bb89_1011cf00d433.slice/crio-7c5f40e0b9e26e253a9e4263bb2d77a058c35397ed6a15a29e4add4f9ca044bc WatchSource:0}: Error finding container 7c5f40e0b9e26e253a9e4263bb2d77a058c35397ed6a15a29e4add4f9ca044bc: Status 404 returned error can't find the container with id 7c5f40e0b9e26e253a9e4263bb2d77a058c35397ed6a15a29e4add4f9ca044bc
Sep 10 23:54:52.104795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.104769    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"]
Sep 10 23:54:52.106664 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:54:52.106640    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1e418b_89a0_430d_b8fa_acece16ee0c4.slice/crio-1e3d84ce7d26a592f711099ef284944f23c8f010cb0a9a48e503bd871e185107 WatchSource:0}: Error finding container 1e3d84ce7d26a592f711099ef284944f23c8f010cb0a9a48e503bd871e185107: Status 404 returned error can't find the container with id 1e3d84ce7d26a592f711099ef284944f23c8f010cb0a9a48e503bd871e185107
Sep 10 23:54:52.831397 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.831359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576" event={"ID":"1aaf3e09-3091-4647-bb89-1011cf00d433","Type":"ContainerStarted","Data":"8a6948f5426dad4be105265c7adcbe7ff4d982bf44a015904b0e51c7219adbf4"}
Sep 10 23:54:52.831821 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.831421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576" event={"ID":"1aaf3e09-3091-4647-bb89-1011cf00d433","Type":"ContainerStarted","Data":"7c5f40e0b9e26e253a9e4263bb2d77a058c35397ed6a15a29e4add4f9ca044bc"}
Sep 10 23:54:52.831821 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.831557    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:54:52.833180 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.833155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh" event={"ID":"53923999-1831-452c-8399-e7e81365abc5","Type":"ContainerStarted","Data":"973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d"}
Sep 10 23:54:52.833310 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.833198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh" event={"ID":"53923999-1831-452c-8399-e7e81365abc5","Type":"ContainerStarted","Data":"b0c48219ded1799edc2b9d81117f56404a60bb0495dabc6c9fb33eb759c194fb"}
Sep 10 23:54:52.833310 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.833227    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:52.834774 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.834749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp" event={"ID":"8e1e418b-89a0-430d-b8fa-acece16ee0c4","Type":"ContainerStarted","Data":"6360344f63928105659f8ad5a2a1337a4b301802530bca7b87e0b05b7cc1d4ac"}
Sep 10 23:54:52.834898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.834794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp" event={"ID":"8e1e418b-89a0-430d-b8fa-acece16ee0c4","Type":"ContainerStarted","Data":"1e3d84ce7d26a592f711099ef284944f23c8f010cb0a9a48e503bd871e185107"}
Sep 10 23:54:52.834898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.834870    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:54:52.847875 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.847829    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576" podStartSLOduration=1.8478163520000002 podStartE2EDuration="1.847816352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:52.847284738 +0000 UTC m=+1104.678312294" watchObservedRunningTime="2026-09-10 23:54:52.847816352 +0000 UTC m=+1104.678843942"
Sep 10 23:54:52.864612 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.864569    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp" podStartSLOduration=1.864555965 podStartE2EDuration="1.864555965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:52.863050581 +0000 UTC m=+1104.694078135" watchObservedRunningTime="2026-09-10 23:54:52.864555965 +0000 UTC m=+1104.695583532"
Sep 10 23:54:52.885872 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:52.885826    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh" podStartSLOduration=2.42348901 podStartE2EDuration="2.885811397s" totalImagesPullingTime="462.322387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:52.885684198 +0000 UTC m=+1104.716711730" watchObservedRunningTime="2026-09-10 23:54:52.885811397 +0000 UTC m=+1104.716838935"
Sep 10 23:54:57.809792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:57.809752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:58.490229 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:58.490193    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:58.572309 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:58.572156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:58.639912 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:58.639876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:58.707812 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:58.707788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:54:58.844276 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:58.844206    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 10 23:54:59.040805 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:54:59.040775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:03.841134 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:03.841105    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 10 23:55:03.841536 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:03.841181    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 10 23:55:05.024557 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:05.024523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:05.124607 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:05.124571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:05.325299 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:05.325215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:05.830542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:05.830510    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:06.164184 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:06.164109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:06.408534 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:06.408504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:09.602247 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:09.602208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:09.707789 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:09.707752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:10.776033 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:10.775997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:10.851145 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:10.851112    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:11.045526 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.045440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:11.362409 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.362325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:11.466334 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.466296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:11.480886 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.480861    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"]
Sep 10 23:55:11.484223 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.484207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:11.486884 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.486856    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d7a539\""
Sep 10 23:55:11.486884 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.486876    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d7a539-dockercfg-mzcdp\""
Sep 10 23:55:11.496587 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.496566    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"]
Sep 10 23:55:11.606221 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.606188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjhc\" (UniqueName: \"kubernetes.io/projected/00daa5cc-4569-4b43-91f1-45f03df24a49-kube-api-access-nbjhc\") pod \"payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f\" (UID: \"00daa5cc-4569-4b43-91f1-45f03df24a49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:11.606380 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.606263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00daa5cc-4569-4b43-91f1-45f03df24a49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f\" (UID: \"00daa5cc-4569-4b43-91f1-45f03df24a49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:11.707743 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.707701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjhc\" (UniqueName: \"kubernetes.io/projected/00daa5cc-4569-4b43-91f1-45f03df24a49-kube-api-access-nbjhc\") pod \"payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f\" (UID: \"00daa5cc-4569-4b43-91f1-45f03df24a49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:11.707945 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.707770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00daa5cc-4569-4b43-91f1-45f03df24a49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f\" (UID: \"00daa5cc-4569-4b43-91f1-45f03df24a49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:11.708429 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.708406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00daa5cc-4569-4b43-91f1-45f03df24a49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f\" (UID: \"00daa5cc-4569-4b43-91f1-45f03df24a49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:11.716173 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.716150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjhc\" (UniqueName: \"kubernetes.io/projected/00daa5cc-4569-4b43-91f1-45f03df24a49-kube-api-access-nbjhc\") pod \"payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f\" (UID: \"00daa5cc-4569-4b43-91f1-45f03df24a49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:11.796602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.796571    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:11.903698 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:11.903665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:12.065665 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:12.065641    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"]
Sep 10 23:55:12.067756 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:55:12.067728    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00daa5cc_4569_4b43_91f1_45f03df24a49.slice/crio-86d53a9fa37fca171e4cb5631976d3550e68ced18f29d8b03fd92e7b76907c8e WatchSource:0}: Error finding container 86d53a9fa37fca171e4cb5631976d3550e68ced18f29d8b03fd92e7b76907c8e: Status 404 returned error can't find the container with id 86d53a9fa37fca171e4cb5631976d3550e68ced18f29d8b03fd92e7b76907c8e
Sep 10 23:55:12.928744 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:12.928710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f" event={"ID":"00daa5cc-4569-4b43-91f1-45f03df24a49","Type":"ContainerStarted","Data":"b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8"}
Sep 10 23:55:12.929142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:12.928770    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f" event={"ID":"00daa5cc-4569-4b43-91f1-45f03df24a49","Type":"ContainerStarted","Data":"86d53a9fa37fca171e4cb5631976d3550e68ced18f29d8b03fd92e7b76907c8e"}
Sep 10 23:55:12.929142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:12.928802    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:12.943038 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:12.943008    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:12.948684 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:12.948644    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f" podStartSLOduration=1.948631487 podStartE2EDuration="1.948631487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:12.947452347 +0000 UTC m=+1124.778479879" watchObservedRunningTime="2026-09-10 23:55:12.948631487 +0000 UTC m=+1124.779659019"
Sep 10 23:55:14.415174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:14.415140    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:14.728997 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:14.728970    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:15.048029 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:15.047954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:15.313428 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:15.313344    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:17.112692 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:17.112654    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:17.213267 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:17.213240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:17.705447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:17.705415    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:17.805661 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:17.805622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:18.958567 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:18.958535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:19.035238 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.035206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:19.254672 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.254593    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:19.436573 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.436541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:19.530522 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.530423    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:19.545649 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.545622    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"]
Sep 10 23:55:19.549279 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.549260    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:19.552242 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.552224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a78403-dockercfg-hz6nn\""
Sep 10 23:55:19.552631 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.552612    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a78403\""
Sep 10 23:55:19.562683 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.562658    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"]
Sep 10 23:55:19.675428 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.675392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thlxz\" (UniqueName: \"kubernetes.io/projected/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-kube-api-access-thlxz\") pod \"payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r\" (UID: \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:19.675428 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.675429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r\" (UID: \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:19.776371 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.776336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thlxz\" (UniqueName: \"kubernetes.io/projected/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-kube-api-access-thlxz\") pod \"payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r\" (UID: \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:19.776545 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.776388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r\" (UID: \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:19.777132 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.777108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r\" (UID: \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:19.784206 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.784146    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thlxz\" (UniqueName: \"kubernetes.io/projected/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-kube-api-access-thlxz\") pod \"payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r\" (UID: \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:19.859197 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:19.859170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:20.137419 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:20.137333    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"]
Sep 10 23:55:20.970365 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:20.970333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r" event={"ID":"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f","Type":"ContainerStarted","Data":"c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272"}
Sep 10 23:55:20.970544 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:20.970382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r" event={"ID":"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f","Type":"ContainerStarted","Data":"95e9395671fefb03d2d65f06737abab8e76abf9d2d838a5c5fb389b39dafccc0"}
Sep 10 23:55:20.970544 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:20.970467    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:20.988428 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:20.988386    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r" podStartSLOduration=1.988373525 podStartE2EDuration="1.988373525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:20.986821461 +0000 UTC m=+1132.817849034" watchObservedRunningTime="2026-09-10 23:55:20.988373525 +0000 UTC m=+1132.819401094"
Sep 10 23:55:23.935151 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:23.935118    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 10 23:55:31.760554 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:31.760520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:31.977494 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:31.977465    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 10 23:55:34.563353 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:34.563291    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:34.910871 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:34.910794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:35.271833 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:35.271800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:35.552068 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:35.551989    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:38.880408 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:38.880355    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:38.977377 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:38.977348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:39.830769 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:39.830732    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:40.714652 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.714620    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"]
Sep 10 23:55:40.719792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.719773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.722215 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.722197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4094ce-serving-cert\""
Sep 10 23:55:40.728431 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.728401    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"]
Sep 10 23:55:40.735879 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.735858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.735983 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.735897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.735983 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.735938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10093fb8-71d1-49b6-aaba-d538110c1ca3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.736076 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.736011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66kzs\" (UniqueName: \"kubernetes.io/projected/10093fb8-71d1-49b6-aaba-d538110c1ca3-kube-api-access-66kzs\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.822862 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.822815    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:40.836936 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.836895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10093fb8-71d1-49b6-aaba-d538110c1ca3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.837072 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.836961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66kzs\" (UniqueName: \"kubernetes.io/projected/10093fb8-71d1-49b6-aaba-d538110c1ca3-kube-api-access-66kzs\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.837072 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.837025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.837072 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.837067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.837248 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:40.837165    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-serving-cert: secret "maas-api-e2e-shared-b-w3-4094ce-serving-cert" not found
Sep 10 23:55:40.837248 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:40.837242    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls podName:10093fb8-71d1-49b6-aaba-d538110c1ca3 nodeName:}" failed. No retries permitted until 2026-09-10 23:55:41.337223924 +0000 UTC m=+1153.168251453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls") pod "maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw" (UID: "10093fb8-71d1-49b6-aaba-d538110c1ca3") : secret "maas-api-e2e-shared-b-w3-4094ce-serving-cert" not found
Sep 10 23:55:40.837602 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.837580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10093fb8-71d1-49b6-aaba-d538110c1ca3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.839475 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.839454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.847569 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.847549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66kzs\" (UniqueName: \"kubernetes.io/projected/10093fb8-71d1-49b6-aaba-d538110c1ca3-kube-api-access-66kzs\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:40.925990 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:40.925961    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:41.089093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.089018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:41.340648 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.340582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:41.342813 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.342791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:41.385278 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.385241    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:41.541027 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.540994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:41.631018 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.630948    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:41.658318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.658285    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"]
Sep 10 23:55:41.662781 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.662760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:41.668757 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.668724    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4094ce\""
Sep 10 23:55:41.669759 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.669730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4094ce-dockercfg-dpcpq\""
Sep 10 23:55:41.689244 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.689221    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"]
Sep 10 23:55:41.744100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.744068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6963e146-5757-40e2-9a78-64e76533402c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl\" (UID: \"6963e146-5757-40e2-9a78-64e76533402c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:41.744100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.744102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtl9b\" (UniqueName: \"kubernetes.io/projected/6963e146-5757-40e2-9a78-64e76533402c-kube-api-access-gtl9b\") pod \"payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl\" (UID: \"6963e146-5757-40e2-9a78-64e76533402c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:41.844758 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.844728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6963e146-5757-40e2-9a78-64e76533402c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl\" (UID: \"6963e146-5757-40e2-9a78-64e76533402c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:41.844944 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.844775    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtl9b\" (UniqueName: \"kubernetes.io/projected/6963e146-5757-40e2-9a78-64e76533402c-kube-api-access-gtl9b\") pod \"payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl\" (UID: \"6963e146-5757-40e2-9a78-64e76533402c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:41.845517 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.845497    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6963e146-5757-40e2-9a78-64e76533402c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl\" (UID: \"6963e146-5757-40e2-9a78-64e76533402c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:41.863168 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.863144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtl9b\" (UniqueName: \"kubernetes.io/projected/6963e146-5757-40e2-9a78-64e76533402c-kube-api-access-gtl9b\") pod \"payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl\" (UID: \"6963e146-5757-40e2-9a78-64e76533402c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:41.914111 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.914054    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"]
Sep 10 23:55:41.917421 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:55:41.917395    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10093fb8_71d1_49b6_aaba_d538110c1ca3.slice/crio-4d5f2920ded81a0393664c575fb92aef757b00eea3b63c722522d5d748a69f08 WatchSource:0}: Error finding container 4d5f2920ded81a0393664c575fb92aef757b00eea3b63c722522d5d748a69f08: Status 404 returned error can't find the container with id 4d5f2920ded81a0393664c575fb92aef757b00eea3b63c722522d5d748a69f08
Sep 10 23:55:41.975045 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:41.975017    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:42.058017 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:42.057989    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:42.072828 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:42.072799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw" event={"ID":"10093fb8-71d1-49b6-aaba-d538110c1ca3","Type":"ContainerStarted","Data":"4d5f2920ded81a0393664c575fb92aef757b00eea3b63c722522d5d748a69f08"}
Sep 10 23:55:42.253711 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:42.253685    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"]
Sep 10 23:55:42.255567 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:55:42.255544    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6963e146_5757_40e2_9a78_64e76533402c.slice/crio-0cb0b9a8c101594c096d313f6f00df8fc8b2293861d2210ace1d866183cbf405 WatchSource:0}: Error finding container 0cb0b9a8c101594c096d313f6f00df8fc8b2293861d2210ace1d866183cbf405: Status 404 returned error can't find the container with id 0cb0b9a8c101594c096d313f6f00df8fc8b2293861d2210ace1d866183cbf405
Sep 10 23:55:42.659726 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:42.659648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:42.794299 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:42.794264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:43.079338 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.079302    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw" event={"ID":"10093fb8-71d1-49b6-aaba-d538110c1ca3","Type":"ContainerStarted","Data":"993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65"}
Sep 10 23:55:43.079528 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.079455    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:43.080917 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.080889    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl" event={"ID":"6963e146-5757-40e2-9a78-64e76533402c","Type":"ContainerStarted","Data":"f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095"}
Sep 10 23:55:43.081054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.080945    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl" event={"ID":"6963e146-5757-40e2-9a78-64e76533402c","Type":"ContainerStarted","Data":"0cb0b9a8c101594c096d313f6f00df8fc8b2293861d2210ace1d866183cbf405"}
Sep 10 23:55:43.081054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.080967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:43.102437 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.102390    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw" podStartSLOduration=2.711985997 podStartE2EDuration="3.102345203s" totalImagesPullingTime="390.359206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:43.101088614 +0000 UTC m=+1154.932116149" watchObservedRunningTime="2026-09-10 23:55:43.102345203 +0000 UTC m=+1154.933372735"
Sep 10 23:55:43.119344 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.119301    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl" podStartSLOduration=2.119286505 podStartE2EDuration="2.119286505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:43.119006011 +0000 UTC m=+1154.950033554" watchObservedRunningTime="2026-09-10 23:55:43.119286505 +0000 UTC m=+1154.950314038"
Sep 10 23:55:43.397754 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.397687    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7689-471b8f"]
Sep 10 23:55:43.401219 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.401203    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:43.403681 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.403653    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nfkgj\""
Sep 10 23:55:43.409269 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.409246    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-471b8f"]
Sep 10 23:55:43.459142 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.459111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpc2\" (UniqueName: \"kubernetes.io/projected/c88b3a8e-22ff-4bef-87b5-d83b79751383-kube-api-access-fmpc2\") pod \"test-curl-7689-471b8f\" (UID: \"c88b3a8e-22ff-4bef-87b5-d83b79751383\") " pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:43.559811 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.559786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpc2\" (UniqueName: \"kubernetes.io/projected/c88b3a8e-22ff-4bef-87b5-d83b79751383-kube-api-access-fmpc2\") pod \"test-curl-7689-471b8f\" (UID: \"c88b3a8e-22ff-4bef-87b5-d83b79751383\") " pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:43.567504 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.567476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpc2\" (UniqueName: \"kubernetes.io/projected/c88b3a8e-22ff-4bef-87b5-d83b79751383-kube-api-access-fmpc2\") pod \"test-curl-7689-471b8f\" (UID: \"c88b3a8e-22ff-4bef-87b5-d83b79751383\") " pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:43.712386 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.712363    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:43.980318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:43.980296    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-471b8f"]
Sep 10 23:55:43.982200 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:55:43.982170    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc88b3a8e_22ff_4bef_87b5_d83b79751383.slice/crio-cf94ffa41148342f3b8521c4075550ee63964dcbcbd05b4bed48c78bce43e0d2 WatchSource:0}: Error finding container cf94ffa41148342f3b8521c4075550ee63964dcbcbd05b4bed48c78bce43e0d2: Status 404 returned error can't find the container with id cf94ffa41148342f3b8521c4075550ee63964dcbcbd05b4bed48c78bce43e0d2
Sep 10 23:55:44.087209 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.087173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-471b8f" event={"ID":"c88b3a8e-22ff-4bef-87b5-d83b79751383","Type":"ContainerStarted","Data":"9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce"}
Sep 10 23:55:44.087319 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.087225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-471b8f" event={"ID":"c88b3a8e-22ff-4bef-87b5-d83b79751383","Type":"ContainerStarted","Data":"cf94ffa41148342f3b8521c4075550ee63964dcbcbd05b4bed48c78bce43e0d2"}
Sep 10 23:55:44.101524 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.101479    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7689-471b8f" podStartSLOduration=1.1014652520000001 podStartE2EDuration="1.101465252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:44.100899411 +0000 UTC m=+1155.931926943" watchObservedRunningTime="2026-09-10 23:55:44.101465252 +0000 UTC m=+1155.932492784"
Sep 10 23:55:44.680463 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.680381    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7689-471b8f"]
Sep 10 23:55:44.684125 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.684100    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7689-471b8f"]
Sep 10 23:55:44.817335 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.817309    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7689-bb93de"]
Sep 10 23:55:44.817728 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.817714    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88b3a8e-22ff-4bef-87b5-d83b79751383" containerName="test-curl-7689-471b8f"
Sep 10 23:55:44.817778 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.817730    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88b3a8e-22ff-4bef-87b5-d83b79751383" containerName="test-curl-7689-471b8f"
Sep 10 23:55:44.817813 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.817796    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88b3a8e-22ff-4bef-87b5-d83b79751383" containerName="test-curl-7689-471b8f"
Sep 10 23:55:44.820895 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.820878    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:44.835532 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.835511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-bb93de"]
Sep 10 23:55:44.872148 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.872123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fqs6\" (UniqueName: \"kubernetes.io/projected/15353ab6-b231-4872-8544-42597e34a270-kube-api-access-8fqs6\") pod \"test-curl-7689-bb93de\" (UID: \"15353ab6-b231-4872-8544-42597e34a270\") " pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:44.973386 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.973362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fqs6\" (UniqueName: \"kubernetes.io/projected/15353ab6-b231-4872-8544-42597e34a270-kube-api-access-8fqs6\") pod \"test-curl-7689-bb93de\" (UID: \"15353ab6-b231-4872-8544-42597e34a270\") " pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:44.983567 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:44.983540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fqs6\" (UniqueName: \"kubernetes.io/projected/15353ab6-b231-4872-8544-42597e34a270-kube-api-access-8fqs6\") pod \"test-curl-7689-bb93de\" (UID: \"15353ab6-b231-4872-8544-42597e34a270\") " pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:45.091781 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:45.091742    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7689-471b8f" container="test-curl-7689-471b8f"
Sep 10 23:55:45.129730 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:45.129703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:45.397536 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:45.397513    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-bb93de"]
Sep 10 23:55:45.399894 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:55:45.399862    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15353ab6_b231_4872_8544_42597e34a270.slice/crio-c05a54b4979eb82305ffb3bbbcf069cc0a57aa4128f2126befe12b58355ec406 WatchSource:0}: Error finding container c05a54b4979eb82305ffb3bbbcf069cc0a57aa4128f2126befe12b58355ec406: Status 404 returned error can't find the container with id c05a54b4979eb82305ffb3bbbcf069cc0a57aa4128f2126befe12b58355ec406
Sep 10 23:55:46.098120 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.098087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-bb93de" event={"ID":"15353ab6-b231-4872-8544-42597e34a270","Type":"ContainerStarted","Data":"16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef"}
Sep 10 23:55:46.098546 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.098129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-bb93de" event={"ID":"15353ab6-b231-4872-8544-42597e34a270","Type":"ContainerStarted","Data":"c05a54b4979eb82305ffb3bbbcf069cc0a57aa4128f2126befe12b58355ec406"}
Sep 10 23:55:46.098546 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.098129    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7689-471b8f" podUID="c88b3a8e-22ff-4bef-87b5-d83b79751383" containerName="test-curl-7689-471b8f" containerID="cri-o://9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce" gracePeriod=2
Sep 10 23:55:46.100361 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:46.100338    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-471b8f\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="c88b3a8e-22ff-4bef-87b5-d83b79751383" pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:46.116481 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.116440    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7689-bb93de" podStartSLOduration=2.116426923 podStartE2EDuration="2.116426923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:46.11518005 +0000 UTC m=+1157.946207583" watchObservedRunningTime="2026-09-10 23:55:46.116426923 +0000 UTC m=+1157.947454480"
Sep 10 23:55:46.117185 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:46.117160    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-471b8f\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="c88b3a8e-22ff-4bef-87b5-d83b79751383" pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:46.743671 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.743626    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7689-bb93de"]
Sep 10 23:55:46.746410 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.746384    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7689-bb93de"]
Sep 10 23:55:46.857919 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.857850    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7689-027334"]
Sep 10 23:55:46.858275 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.858261    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15353ab6-b231-4872-8544-42597e34a270" containerName="test-curl-7689-bb93de"
Sep 10 23:55:46.858318 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.858276    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="15353ab6-b231-4872-8544-42597e34a270" containerName="test-curl-7689-bb93de"
Sep 10 23:55:46.858358 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.858350    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="15353ab6-b231-4872-8544-42597e34a270" containerName="test-curl-7689-bb93de"
Sep 10 23:55:46.861542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.861516    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:46.866224 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.866201    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-027334"]
Sep 10 23:55:46.892565 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.892539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nn7\" (UniqueName: \"kubernetes.io/projected/9c431ea3-48da-42ab-a5ff-ce95d8374195-kube-api-access-z5nn7\") pod \"test-curl-7689-027334\" (UID: \"9c431ea3-48da-42ab-a5ff-ce95d8374195\") " pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:46.993547 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:46.993522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nn7\" (UniqueName: \"kubernetes.io/projected/9c431ea3-48da-42ab-a5ff-ce95d8374195-kube-api-access-z5nn7\") pod \"test-curl-7689-027334\" (UID: \"9c431ea3-48da-42ab-a5ff-ce95d8374195\") " pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:47.005397 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:47.005289    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nn7\" (UniqueName: \"kubernetes.io/projected/9c431ea3-48da-42ab-a5ff-ce95d8374195-kube-api-access-z5nn7\") pod \"test-curl-7689-027334\" (UID: \"9c431ea3-48da-42ab-a5ff-ce95d8374195\") " pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:47.102887 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:47.102855    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7689-bb93de" container="test-curl-7689-bb93de"
Sep 10 23:55:47.172174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:47.172115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:47.439192 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:47.439165    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-027334"]
Sep 10 23:55:47.441356 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:55:47.441328    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c431ea3_48da_42ab_a5ff_ce95d8374195.slice/crio-f74bfe85ea70778c438d0ea674d96067d6a9bec7eae07e9960ba59038fe09250 WatchSource:0}: Error finding container f74bfe85ea70778c438d0ea674d96067d6a9bec7eae07e9960ba59038fe09250: Status 404 returned error can't find the container with id f74bfe85ea70778c438d0ea674d96067d6a9bec7eae07e9960ba59038fe09250
Sep 10 23:55:48.110801 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.110769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-027334" event={"ID":"9c431ea3-48da-42ab-a5ff-ce95d8374195","Type":"ContainerStarted","Data":"146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2"}
Sep 10 23:55:48.111220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.110818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-027334" event={"ID":"9c431ea3-48da-42ab-a5ff-ce95d8374195","Type":"ContainerStarted","Data":"f74bfe85ea70778c438d0ea674d96067d6a9bec7eae07e9960ba59038fe09250"}
Sep 10 23:55:48.111220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.110826    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7689-bb93de" podUID="15353ab6-b231-4872-8544-42597e34a270" containerName="test-curl-7689-bb93de" containerID="cri-o://16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef" gracePeriod=2
Sep 10 23:55:48.113269 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:48.113233    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-bb93de\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="15353ab6-b231-4872-8544-42597e34a270" pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:48.128555 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.128507    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7689-027334" podStartSLOduration=2.128493397 podStartE2EDuration="2.128493397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:48.127295388 +0000 UTC m=+1159.958322927" watchObservedRunningTime="2026-09-10 23:55:48.128493397 +0000 UTC m=+1159.959520979"
Sep 10 23:55:48.129188 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:48.129144    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-bb93de\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="15353ab6-b231-4872-8544-42597e34a270" pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:48.249019 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.248991    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:48.251639 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:48.251612    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-bb93de\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="15353ab6-b231-4872-8544-42597e34a270" pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:48.304214 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.304187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmpc2\" (UniqueName: \"kubernetes.io/projected/c88b3a8e-22ff-4bef-87b5-d83b79751383-kube-api-access-fmpc2\") pod \"c88b3a8e-22ff-4bef-87b5-d83b79751383\" (UID: \"c88b3a8e-22ff-4bef-87b5-d83b79751383\") "
Sep 10 23:55:48.306291 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.306264    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88b3a8e-22ff-4bef-87b5-d83b79751383-kube-api-access-fmpc2" (OuterVolumeSpecName: "kube-api-access-fmpc2") pod "c88b3a8e-22ff-4bef-87b5-d83b79751383" (UID: "c88b3a8e-22ff-4bef-87b5-d83b79751383"). InnerVolumeSpecName "kube-api-access-fmpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:55:48.405439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.405357    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmpc2\" (UniqueName: \"kubernetes.io/projected/c88b3a8e-22ff-4bef-87b5-d83b79751383-kube-api-access-fmpc2\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:55:48.654441 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:48.654406    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-bb93de\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="15353ab6-b231-4872-8544-42597e34a270" pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:48.656730 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.656673    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88b3a8e-22ff-4bef-87b5-d83b79751383" path="/var/lib/kubelet/pods/c88b3a8e-22ff-4bef-87b5-d83b79751383/volumes"
Sep 10 23:55:48.706385 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.706359    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7689-027334"]
Sep 10 23:55:48.712795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.712769    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7689-027334"]
Sep 10 23:55:48.820973 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.820947    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7689-532331"]
Sep 10 23:55:48.821354 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.821342    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c431ea3-48da-42ab-a5ff-ce95d8374195" containerName="test-curl-7689-027334"
Sep 10 23:55:48.821398 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.821356    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c431ea3-48da-42ab-a5ff-ce95d8374195" containerName="test-curl-7689-027334"
Sep 10 23:55:48.821439 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.821416    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c431ea3-48da-42ab-a5ff-ce95d8374195" containerName="test-curl-7689-027334"
Sep 10 23:55:48.824492 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.824477    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:48.830279 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.830254    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-532331"]
Sep 10 23:55:48.909668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:48.909591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njcxw\" (UniqueName: \"kubernetes.io/projected/6df856ca-3079-43ce-af01-2dd113c9dee7-kube-api-access-njcxw\") pod \"test-curl-7689-532331\" (UID: \"6df856ca-3079-43ce-af01-2dd113c9dee7\") " pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:49.010450 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.010422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcxw\" (UniqueName: \"kubernetes.io/projected/6df856ca-3079-43ce-af01-2dd113c9dee7-kube-api-access-njcxw\") pod \"test-curl-7689-532331\" (UID: \"6df856ca-3079-43ce-af01-2dd113c9dee7\") " pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:49.018525 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.018496    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcxw\" (UniqueName: \"kubernetes.io/projected/6df856ca-3079-43ce-af01-2dd113c9dee7-kube-api-access-njcxw\") pod \"test-curl-7689-532331\" (UID: \"6df856ca-3079-43ce-af01-2dd113c9dee7\") " pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:49.092687 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.092665    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 10 23:55:49.116248 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.116220    2562 generic.go:356] "Generic (PLEG): container finished" podID="c88b3a8e-22ff-4bef-87b5-d83b79751383" containerID="9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce" exitCode=137
Sep 10 23:55:49.116736 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.116269    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-471b8f"
Sep 10 23:55:49.116736 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.116318    2562 scope.go:117] "RemoveContainer" containerID="9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce"
Sep 10 23:55:49.116736 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:49.116659    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7689-027334" container="test-curl-7689-027334"
Sep 10 23:55:49.130894 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.130870    2562 scope.go:117] "RemoveContainer" containerID="9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce"
Sep 10 23:55:49.131240 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:49.131219    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce\": container with ID starting with 9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce not found: ID does not exist" containerID="9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce"
Sep 10 23:55:49.131339 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.131250    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce"} err="failed to get container status \"9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce\": rpc error: code = NotFound desc = could not find container \"9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce\": container with ID starting with 9e5fcc45b4aa136c473433a508b2aee02cd060ba08def5310263a70de499f7ce not found: ID does not exist"
Sep 10 23:55:49.134320 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.134297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:49.413288 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:49.413252    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-532331"]
Sep 10 23:55:50.123201 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.123167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-532331" event={"ID":"6df856ca-3079-43ce-af01-2dd113c9dee7","Type":"ContainerStarted","Data":"457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa"}
Sep 10 23:55:50.123517 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.123214    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-532331" event={"ID":"6df856ca-3079-43ce-af01-2dd113c9dee7","Type":"ContainerStarted","Data":"2bf13320954b6797bb383bf7f73b309b3f7a2f8d23e3f4c046ad142dac98a940"}
Sep 10 23:55:50.123517 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.123345    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7689-027334" podUID="9c431ea3-48da-42ab-a5ff-ce95d8374195" containerName="test-curl-7689-027334" containerID="cri-o://146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2" gracePeriod=2
Sep 10 23:55:50.137201 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.137149    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7689-532331" podStartSLOduration=2.137135596 podStartE2EDuration="2.137135596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:50.136237636 +0000 UTC m=+1161.967265172" watchObservedRunningTime="2026-09-10 23:55:50.137135596 +0000 UTC m=+1161.968163158"
Sep 10 23:55:50.138080 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:50.138055    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-027334\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="9c431ea3-48da-42ab-a5ff-ce95d8374195" pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:50.285885 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.285864    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:50.288111 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:50.288087    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-027334\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="9c431ea3-48da-42ab-a5ff-ce95d8374195" pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:50.322422 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.322398    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fqs6\" (UniqueName: \"kubernetes.io/projected/15353ab6-b231-4872-8544-42597e34a270-kube-api-access-8fqs6\") pod \"15353ab6-b231-4872-8544-42597e34a270\" (UID: \"15353ab6-b231-4872-8544-42597e34a270\") "
Sep 10 23:55:50.324411 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.324385    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15353ab6-b231-4872-8544-42597e34a270-kube-api-access-8fqs6" (OuterVolumeSpecName: "kube-api-access-8fqs6") pod "15353ab6-b231-4872-8544-42597e34a270" (UID: "15353ab6-b231-4872-8544-42597e34a270"). InnerVolumeSpecName "kube-api-access-8fqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:55:50.423551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.423494    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fqs6\" (UniqueName: \"kubernetes.io/projected/15353ab6-b231-4872-8544-42597e34a270-kube-api-access-8fqs6\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:55:50.660245 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.660206    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15353ab6-b231-4872-8544-42597e34a270" path="/var/lib/kubelet/pods/15353ab6-b231-4872-8544-42597e34a270/volumes"
Sep 10 23:55:50.719879 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.719850    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7689-532331"]
Sep 10 23:55:50.722377 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:50.722355    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7689-532331"]
Sep 10 23:55:51.129472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:51.129356    2562 generic.go:356] "Generic (PLEG): container finished" podID="15353ab6-b231-4872-8544-42597e34a270" containerID="16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef" exitCode=137
Sep 10 23:55:51.129472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:51.129416    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-bb93de"
Sep 10 23:55:51.129472 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:51.129456    2562 scope.go:117] "RemoveContainer" containerID="16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef"
Sep 10 23:55:51.129946 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:51.129633    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7689-532331" container="test-curl-7689-532331"
Sep 10 23:55:51.137699 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:51.137682    2562 scope.go:117] "RemoveContainer" containerID="16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef"
Sep 10 23:55:51.137999 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:51.137978    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef\": container with ID starting with 16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef not found: ID does not exist" containerID="16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef"
Sep 10 23:55:51.138054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:51.138005    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef"} err="failed to get container status \"16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef\": rpc error: code = NotFound desc = could not find container \"16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef\": container with ID starting with 16f4559f75843eba52a4cff195d1c9de7b62c7ef0c3c027bd47f068dfdbbb1ef not found: ID does not exist"
Sep 10 23:55:51.293423 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:51.293396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:51.596819 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:51.596791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:51.921065 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:51.920987    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:52.136133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:52.136071    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7689-532331" podUID="6df856ca-3079-43ce-af01-2dd113c9dee7" containerName="test-curl-7689-532331" containerID="cri-o://457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa" gracePeriod=2
Sep 10 23:55:52.138613 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:52.138589    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-532331\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="6df856ca-3079-43ce-af01-2dd113c9dee7" pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:52.269656 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:52.269630    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:52.271830 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:52.271807    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7689-532331\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="6df856ca-3079-43ce-af01-2dd113c9dee7" pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:52.337416 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:52.337389    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5nn7\" (UniqueName: \"kubernetes.io/projected/9c431ea3-48da-42ab-a5ff-ce95d8374195-kube-api-access-z5nn7\") pod \"9c431ea3-48da-42ab-a5ff-ce95d8374195\" (UID: \"9c431ea3-48da-42ab-a5ff-ce95d8374195\") "
Sep 10 23:55:52.339512 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:52.339489    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c431ea3-48da-42ab-a5ff-ce95d8374195-kube-api-access-z5nn7" (OuterVolumeSpecName: "kube-api-access-z5nn7") pod "9c431ea3-48da-42ab-a5ff-ce95d8374195" (UID: "9c431ea3-48da-42ab-a5ff-ce95d8374195"). InnerVolumeSpecName "kube-api-access-z5nn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:55:52.438456 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:52.438427    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5nn7\" (UniqueName: \"kubernetes.io/projected/9c431ea3-48da-42ab-a5ff-ce95d8374195-kube-api-access-z5nn7\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:55:52.656074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:52.656009    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c431ea3-48da-42ab-a5ff-ce95d8374195" path="/var/lib/kubelet/pods/9c431ea3-48da-42ab-a5ff-ce95d8374195/volumes"
Sep 10 23:55:53.141862 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:53.141827    2562 generic.go:356] "Generic (PLEG): container finished" podID="9c431ea3-48da-42ab-a5ff-ce95d8374195" containerID="146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2" exitCode=137
Sep 10 23:55:53.142339 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:53.141869    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-027334"
Sep 10 23:55:53.142339 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:53.141887    2562 scope.go:117] "RemoveContainer" containerID="146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2"
Sep 10 23:55:53.150638 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:53.150595    2562 scope.go:117] "RemoveContainer" containerID="146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2"
Sep 10 23:55:53.150872 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:53.150853    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2\": container with ID starting with 146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2 not found: ID does not exist" containerID="146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2"
Sep 10 23:55:53.151002 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:53.150885    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2"} err="failed to get container status \"146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2\": rpc error: code = NotFound desc = could not find container \"146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2\": container with ID starting with 146b9a88f2be2a8d66f8474bb9359f2c031930a6536c08e02c5a833371e128f2 not found: ID does not exist"
Sep 10 23:55:54.089293 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:54.089267    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 10 23:55:54.314506 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:54.314485    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:54.357275 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:54.357222    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njcxw\" (UniqueName: \"kubernetes.io/projected/6df856ca-3079-43ce-af01-2dd113c9dee7-kube-api-access-njcxw\") pod \"6df856ca-3079-43ce-af01-2dd113c9dee7\" (UID: \"6df856ca-3079-43ce-af01-2dd113c9dee7\") "
Sep 10 23:55:54.359060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:54.359039    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df856ca-3079-43ce-af01-2dd113c9dee7-kube-api-access-njcxw" (OuterVolumeSpecName: "kube-api-access-njcxw") pod "6df856ca-3079-43ce-af01-2dd113c9dee7" (UID: "6df856ca-3079-43ce-af01-2dd113c9dee7"). InnerVolumeSpecName "kube-api-access-njcxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:55:54.458670 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:54.458640    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njcxw\" (UniqueName: \"kubernetes.io/projected/6df856ca-3079-43ce-af01-2dd113c9dee7-kube-api-access-njcxw\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:55:54.655940 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:54.655854    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df856ca-3079-43ce-af01-2dd113c9dee7" path="/var/lib/kubelet/pods/6df856ca-3079-43ce-af01-2dd113c9dee7/volumes"
Sep 10 23:55:54.933470 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:54.933441    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:55.029516 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:55.029488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:55.163226 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:55.163192    2562 generic.go:356] "Generic (PLEG): container finished" podID="6df856ca-3079-43ce-af01-2dd113c9dee7" containerID="457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa" exitCode=137
Sep 10 23:55:55.163385 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:55.163250    2562 scope.go:117] "RemoveContainer" containerID="457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa"
Sep 10 23:55:55.163385 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:55.163255    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-532331"
Sep 10 23:55:55.171780 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:55.171761    2562 scope.go:117] "RemoveContainer" containerID="457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa"
Sep 10 23:55:55.172073 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:55:55.172052    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa\": container with ID starting with 457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa not found: ID does not exist" containerID="457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa"
Sep 10 23:55:55.172149 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:55.172081    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa"} err="failed to get container status \"457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa\": rpc error: code = NotFound desc = could not find container \"457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa\": container with ID starting with 457f5a169029cbcf9b4092c7e53059a934930e60c070684045800ed5542741fa not found: ID does not exist"
Sep 10 23:55:56.137016 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.136980    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"]
Sep 10 23:55:56.137447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.137430    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df856ca-3079-43ce-af01-2dd113c9dee7" containerName="test-curl-7689-532331"
Sep 10 23:55:56.137447 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.137449    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df856ca-3079-43ce-af01-2dd113c9dee7" containerName="test-curl-7689-532331"
Sep 10 23:55:56.137551 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.137533    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df856ca-3079-43ce-af01-2dd113c9dee7" containerName="test-curl-7689-532331"
Sep 10 23:55:56.141750 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.141733    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.144170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.144151    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e1f72544-serving-cert\""
Sep 10 23:55:56.151149 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.151124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"]
Sep 10 23:55:56.172392 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.172360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:56.272848 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.272816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5795bc78-ac04-48f7-9302-139e84fb2915-openshift-service-ca\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.273025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.272882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-maas-api-tls\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.273025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.272943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7s6j\" (UniqueName: \"kubernetes.io/projected/5795bc78-ac04-48f7-9302-139e84fb2915-kube-api-access-h7s6j\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.273025 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.272967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-metrics-certs\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.276030 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.276006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:56.335153 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.335128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:56.374296 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.374263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5795bc78-ac04-48f7-9302-139e84fb2915-openshift-service-ca\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.374397 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.374342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-maas-api-tls\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.374397 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.374368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7s6j\" (UniqueName: \"kubernetes.io/projected/5795bc78-ac04-48f7-9302-139e84fb2915-kube-api-access-h7s6j\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.374397 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.374394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-metrics-certs\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.374859 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.374842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5795bc78-ac04-48f7-9302-139e84fb2915-openshift-service-ca\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.376786 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.376766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-maas-api-tls\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.376894 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.376876    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-metrics-certs\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.383128 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.383100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7s6j\" (UniqueName: \"kubernetes.io/projected/5795bc78-ac04-48f7-9302-139e84fb2915-kube-api-access-h7s6j\") pod \"maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.456733 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.456710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:56.659466 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.659435    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:56.729074 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.729017    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"]
Sep 10 23:55:56.731189 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:55:56.731162    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5795bc78_ac04_48f7_9302_139e84fb2915.slice/crio-6385a49d797d7dbff4585e69c9879c8b8f826cfc008823d88e4d94a7986a478a WatchSource:0}: Error finding container 6385a49d797d7dbff4585e69c9879c8b8f826cfc008823d88e4d94a7986a478a: Status 404 returned error can't find the container with id 6385a49d797d7dbff4585e69c9879c8b8f826cfc008823d88e4d94a7986a478a
Sep 10 23:55:56.733478 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:56.733452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:55:57.176169 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:57.176138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml" event={"ID":"5795bc78-ac04-48f7-9302-139e84fb2915","Type":"ContainerStarted","Data":"6385a49d797d7dbff4585e69c9879c8b8f826cfc008823d88e4d94a7986a478a"}
Sep 10 23:55:58.182096 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:58.182064    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml" event={"ID":"5795bc78-ac04-48f7-9302-139e84fb2915","Type":"ContainerStarted","Data":"cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b"}
Sep 10 23:55:58.182453 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:58.182121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:55:58.201114 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:55:58.201067    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml" podStartSLOduration=1.7692574730000001 podStartE2EDuration="2.201053632s" totalImagesPullingTime="431.796159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:58.198612738 +0000 UTC m=+1170.029640301" watchObservedRunningTime="2026-09-10 23:55:58.201053632 +0000 UTC m=+1170.032081164"
Sep 10 23:56:02.130853 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:02.130816    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:03.840574 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:03.840540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:04.192359 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:04.192333    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:56:13.148609 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:13.148578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:13.646213 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:13.646185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:13.747660 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:13.747627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:14.214059 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.214028    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"]
Sep 10 23:56:14.221107 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.221077    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.224610 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.224585    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"kube-root-ca.crt\""
Sep 10 23:56:14.226512 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.226492    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"auth-test-model-d7a539-kserve-self-signed-certs\""
Sep 10 23:56:14.227139 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.227084    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"openshift-service-ca.crt\""
Sep 10 23:56:14.227435 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.227093    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"]
Sep 10 23:56:14.227540 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.227198    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"default-dockercfg-2scc9\""
Sep 10 23:56:14.253361 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.253338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:14.327288 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.327263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tmp-dir\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.327386 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.327290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tls-certs\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.327386 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.327319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-home\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.327386 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.327362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-model-cache\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.327522 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.327420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-dshm\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.327522 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.327444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j489t\" (UniqueName: \"kubernetes.io/projected/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-kube-api-access-j489t\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428285 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-model-cache\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428392 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-dshm\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428392 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j489t\" (UniqueName: \"kubernetes.io/projected/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-kube-api-access-j489t\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428392 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tmp-dir\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428392 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428376    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tls-certs\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428608 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-home\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428803 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-model-cache\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428874 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tmp-dir\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.428874 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.428841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-home\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.430513 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.430488    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-dshm\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.430844 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.430823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tls-certs\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.435799 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.435779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j489t\" (UniqueName: \"kubernetes.io/projected/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-kube-api-access-j489t\") pod \"auth-test-model-d7a539-kserve-7995686c99-bcmbm\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.535779 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.535720    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:14.805000 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:14.804974    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"]
Sep 10 23:56:14.807873 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:56:14.807850    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ffd6746_3b05_4aeb_9bd4_9af8ed84892e.slice/crio-d00d41014bf71d2bcf35a5a34ba257cfa79547a12f1b9717fe780f42a8e45523 WatchSource:0}: Error finding container d00d41014bf71d2bcf35a5a34ba257cfa79547a12f1b9717fe780f42a8e45523: Status 404 returned error can't find the container with id d00d41014bf71d2bcf35a5a34ba257cfa79547a12f1b9717fe780f42a8e45523
Sep 10 23:56:15.267712 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:15.267681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm" event={"ID":"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e","Type":"ContainerStarted","Data":"18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5"}
Sep 10 23:56:15.268188 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:15.267720    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm" event={"ID":"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e","Type":"ContainerStarted","Data":"d00d41014bf71d2bcf35a5a34ba257cfa79547a12f1b9717fe780f42a8e45523"}
Sep 10 23:56:15.285995 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:15.285952    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm" podStartSLOduration=1.285939232 podStartE2EDuration="1.285939232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:15.284103208 +0000 UTC m=+1187.115130745" watchObservedRunningTime="2026-09-10 23:56:15.285939232 +0000 UTC m=+1187.116966797"
Sep 10 23:56:17.037133 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:17.037096    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:19.875033 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:19.875003    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:24.536829 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:24.536792    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:24.537215 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:24.536845    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:24.549022 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:24.549000    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:25.327581 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:25.327550    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:56:28.632020 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:28.631992    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:56:28.636455 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:28.636433    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 10 23:56:31.222256 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.222222    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"]
Sep 10 23:56:31.227581 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.227556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.231037 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.231005    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1f72544\"/\"kube-root-ca.crt\""
Sep 10 23:56:31.231037 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.231010    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1f72544\"/\"default-dockercfg-l252w\""
Sep 10 23:56:31.231238 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.231043    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1f72544\"/\"openshift-service-ca.crt\""
Sep 10 23:56:31.231238 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.231065    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1f72544\"/\"e2e-lifecycle-model-e1f72544-kserve-self-signed-certs\""
Sep 10 23:56:31.236855 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.236828    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"]
Sep 10 23:56:31.251087 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.251064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:31.263438 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.263413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7a9228-9f50-4a01-8cba-125681b964d7-tls-certs\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.263537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.263454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-home\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.263537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.263512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-model-cache\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.263609 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.263539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-tmp-dir\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.263660 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.263642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkz89\" (UniqueName: \"kubernetes.io/projected/aa7a9228-9f50-4a01-8cba-125681b964d7-kube-api-access-pkz89\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.263700 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.263674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-dshm\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.364986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.364950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkz89\" (UniqueName: \"kubernetes.io/projected/aa7a9228-9f50-4a01-8cba-125681b964d7-kube-api-access-pkz89\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.364986 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.364991    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-dshm\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.365220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.365051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7a9228-9f50-4a01-8cba-125681b964d7-tls-certs\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.365220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.365079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-home\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.365220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.365114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-model-cache\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.365220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.365140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-tmp-dir\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.365537 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.365515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-home\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.365605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.365544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-model-cache\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.365605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.365550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-tmp-dir\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.367313 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.367293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-dshm\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.367559 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.367543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7a9228-9f50-4a01-8cba-125681b964d7-tls-certs\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.372337 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.372316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkz89\" (UniqueName: \"kubernetes.io/projected/aa7a9228-9f50-4a01-8cba-125681b964d7-kube-api-access-pkz89\") pod \"e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") " pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.539208 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.539119    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:31.810585 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:31.810560    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"]
Sep 10 23:56:31.812415 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:56:31.812392    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa7a9228_9f50_4a01_8cba_125681b964d7.slice/crio-84ce7be6dfb2db6ab7ce5513b30fd7e69b77a7a0c83b199fc37ab4fc1145d9be WatchSource:0}: Error finding container 84ce7be6dfb2db6ab7ce5513b30fd7e69b77a7a0c83b199fc37ab4fc1145d9be: Status 404 returned error can't find the container with id 84ce7be6dfb2db6ab7ce5513b30fd7e69b77a7a0c83b199fc37ab4fc1145d9be
Sep 10 23:56:32.351313 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:32.351273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x" event={"ID":"aa7a9228-9f50-4a01-8cba-125681b964d7","Type":"ContainerStarted","Data":"45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f"}
Sep 10 23:56:32.351313 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:32.351319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x" event={"ID":"aa7a9228-9f50-4a01-8cba-125681b964d7","Type":"ContainerStarted","Data":"84ce7be6dfb2db6ab7ce5513b30fd7e69b77a7a0c83b199fc37ab4fc1145d9be"}
Sep 10 23:56:32.368682 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:32.368636    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x" podStartSLOduration=1.3686211209999999 podStartE2EDuration="1.368621121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:32.366860555 +0000 UTC m=+1204.197888101" watchObservedRunningTime="2026-09-10 23:56:32.368621121 +0000 UTC m=+1204.199648655"
Sep 10 23:56:37.415938 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:37.415903    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:56:41.539835 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:41.539803    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:41.540220 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:41.539853    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:41.552651 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:41.552630    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:56:42.409641 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:56:42.409609    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:57:15.061597 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.061565    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"]
Sep 10 23:57:15.062120 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.061838    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml" podUID="5795bc78-ac04-48f7-9302-139e84fb2915" containerName="maas-api" containerID="cri-o://cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b" gracePeriod=30
Sep 10 23:57:15.139430 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.139396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:15.328402 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.328346    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:57:15.443231 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.443204    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-metrics-certs\") pod \"5795bc78-ac04-48f7-9302-139e84fb2915\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") "
Sep 10 23:57:15.443395 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.443256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-maas-api-tls\") pod \"5795bc78-ac04-48f7-9302-139e84fb2915\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") "
Sep 10 23:57:15.443395 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.443341    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7s6j\" (UniqueName: \"kubernetes.io/projected/5795bc78-ac04-48f7-9302-139e84fb2915-kube-api-access-h7s6j\") pod \"5795bc78-ac04-48f7-9302-139e84fb2915\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") "
Sep 10 23:57:15.443395 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.443365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5795bc78-ac04-48f7-9302-139e84fb2915-openshift-service-ca\") pod \"5795bc78-ac04-48f7-9302-139e84fb2915\" (UID: \"5795bc78-ac04-48f7-9302-139e84fb2915\") "
Sep 10 23:57:15.443793 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.443765    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5795bc78-ac04-48f7-9302-139e84fb2915-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5795bc78-ac04-48f7-9302-139e84fb2915" (UID: "5795bc78-ac04-48f7-9302-139e84fb2915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:57:15.445316 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.445287    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5795bc78-ac04-48f7-9302-139e84fb2915" (UID: "5795bc78-ac04-48f7-9302-139e84fb2915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:57:15.445407 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.445365    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5795bc78-ac04-48f7-9302-139e84fb2915-kube-api-access-h7s6j" (OuterVolumeSpecName: "kube-api-access-h7s6j") pod "5795bc78-ac04-48f7-9302-139e84fb2915" (UID: "5795bc78-ac04-48f7-9302-139e84fb2915"). InnerVolumeSpecName "kube-api-access-h7s6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:57:15.445449 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.445405    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5795bc78-ac04-48f7-9302-139e84fb2915" (UID: "5795bc78-ac04-48f7-9302-139e84fb2915"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:57:15.544334 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.544303    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7s6j\" (UniqueName: \"kubernetes.io/projected/5795bc78-ac04-48f7-9302-139e84fb2915-kube-api-access-h7s6j\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:15.544334 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.544331    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5795bc78-ac04-48f7-9302-139e84fb2915-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:15.544493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.544341    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:15.544493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.544351    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5795bc78-ac04-48f7-9302-139e84fb2915-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:15.562829 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.562792    2562 generic.go:356] "Generic (PLEG): container finished" podID="5795bc78-ac04-48f7-9302-139e84fb2915" containerID="cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b" exitCode=0
Sep 10 23:57:15.562945 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.562846    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"
Sep 10 23:57:15.562945 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.562878    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml" event={"ID":"5795bc78-ac04-48f7-9302-139e84fb2915","Type":"ContainerDied","Data":"cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b"}
Sep 10 23:57:15.562945 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.562939    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml" event={"ID":"5795bc78-ac04-48f7-9302-139e84fb2915","Type":"ContainerDied","Data":"6385a49d797d7dbff4585e69c9879c8b8f826cfc008823d88e4d94a7986a478a"}
Sep 10 23:57:15.563062 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.562960    2562 scope.go:117] "RemoveContainer" containerID="cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b"
Sep 10 23:57:15.572098 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.572082    2562 scope.go:117] "RemoveContainer" containerID="cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b"
Sep 10 23:57:15.572346 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:57:15.572329    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b\": container with ID starting with cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b not found: ID does not exist" containerID="cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b"
Sep 10 23:57:15.572391 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.572353    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b"} err="failed to get container status \"cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b\": rpc error: code = NotFound desc = could not find container \"cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b\": container with ID starting with cc1772cb15994a66ce39cc95a379ef62e825a0d736b65d7a2d5de604bd72289b not found: ID does not exist"
Sep 10 23:57:15.602445 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.602392    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"]
Sep 10 23:57:15.610216 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:15.610196    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1f72544-54bb5644d4-dw7ml"]
Sep 10 23:57:16.527200 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:16.527166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:16.656125 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:16.656090    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5795bc78-ac04-48f7-9302-139e84fb2915" path="/var/lib/kubelet/pods/5795bc78-ac04-48f7-9302-139e84fb2915/volumes"
Sep 10 23:57:18.268428 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:18.268396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:18.395604 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:18.395574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:23.370825 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:23.370745    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"]
Sep 10 23:57:23.466628 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:23.466598    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:38.390367 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.390324    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x" podUID="aa7a9228-9f50-4a01-8cba-125681b964d7" containerName="main" containerID="cri-o://45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f" gracePeriod=45
Sep 10 23:57:38.641756 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.641704    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:57:38.677484 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.677451    2562 generic.go:356] "Generic (PLEG): container finished" podID="aa7a9228-9f50-4a01-8cba-125681b964d7" containerID="45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f" exitCode=0
Sep 10 23:57:38.677600 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.677496    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"
Sep 10 23:57:38.677600 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.677504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x" event={"ID":"aa7a9228-9f50-4a01-8cba-125681b964d7","Type":"ContainerDied","Data":"45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f"}
Sep 10 23:57:38.677680 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.677627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x" event={"ID":"aa7a9228-9f50-4a01-8cba-125681b964d7","Type":"ContainerDied","Data":"84ce7be6dfb2db6ab7ce5513b30fd7e69b77a7a0c83b199fc37ab4fc1145d9be"}
Sep 10 23:57:38.677680 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.677656    2562 scope.go:117] "RemoveContainer" containerID="45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f"
Sep 10 23:57:38.686611 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.686595    2562 scope.go:117] "RemoveContainer" containerID="45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f"
Sep 10 23:57:38.686936 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:57:38.686901    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f\": container with ID starting with 45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f not found: ID does not exist" containerID="45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f"
Sep 10 23:57:38.687054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.686953    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f"} err="failed to get container status \"45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f\": rpc error: code = NotFound desc = could not find container \"45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f\": container with ID starting with 45c05959d6452ee9b1f76ea2fdaa72bf6fc35b7463c8273eec25422bbd64448f not found: ID does not exist"
Sep 10 23:57:38.745569 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.745548    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-dshm\") pod \"aa7a9228-9f50-4a01-8cba-125681b964d7\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") "
Sep 10 23:57:38.745675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.745580    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-home\") pod \"aa7a9228-9f50-4a01-8cba-125681b964d7\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") "
Sep 10 23:57:38.745675 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.745619    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkz89\" (UniqueName: \"kubernetes.io/projected/aa7a9228-9f50-4a01-8cba-125681b964d7-kube-api-access-pkz89\") pod \"aa7a9228-9f50-4a01-8cba-125681b964d7\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") "
Sep 10 23:57:38.745795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.745678    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-tmp-dir\") pod \"aa7a9228-9f50-4a01-8cba-125681b964d7\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") "
Sep 10 23:57:38.745795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.745706    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-model-cache\") pod \"aa7a9228-9f50-4a01-8cba-125681b964d7\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") "
Sep 10 23:57:38.745795 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.745730    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7a9228-9f50-4a01-8cba-125681b964d7-tls-certs\") pod \"aa7a9228-9f50-4a01-8cba-125681b964d7\" (UID: \"aa7a9228-9f50-4a01-8cba-125681b964d7\") "
Sep 10 23:57:38.745972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.745897    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-home" (OuterVolumeSpecName: "home") pod "aa7a9228-9f50-4a01-8cba-125681b964d7" (UID: "aa7a9228-9f50-4a01-8cba-125681b964d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:57:38.746046 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.746004    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa7a9228-9f50-4a01-8cba-125681b964d7" (UID: "aa7a9228-9f50-4a01-8cba-125681b964d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:57:38.746093 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.746045    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa7a9228-9f50-4a01-8cba-125681b964d7" (UID: "aa7a9228-9f50-4a01-8cba-125681b964d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:57:38.746143 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.746104    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:38.746143 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.746124    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:38.747721 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.747699    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7a9228-9f50-4a01-8cba-125681b964d7-kube-api-access-pkz89" (OuterVolumeSpecName: "kube-api-access-pkz89") pod "aa7a9228-9f50-4a01-8cba-125681b964d7" (UID: "aa7a9228-9f50-4a01-8cba-125681b964d7"). InnerVolumeSpecName "kube-api-access-pkz89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:57:38.747846 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.747824    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7a9228-9f50-4a01-8cba-125681b964d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa7a9228-9f50-4a01-8cba-125681b964d7" (UID: "aa7a9228-9f50-4a01-8cba-125681b964d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:57:38.747911 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.747845    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-dshm" (OuterVolumeSpecName: "dshm") pod "aa7a9228-9f50-4a01-8cba-125681b964d7" (UID: "aa7a9228-9f50-4a01-8cba-125681b964d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:57:38.846898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.846877    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:38.846898 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.846898    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7a9228-9f50-4a01-8cba-125681b964d7-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:38.847066 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.846907    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7a9228-9f50-4a01-8cba-125681b964d7-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:38.847066 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.846915    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkz89\" (UniqueName: \"kubernetes.io/projected/aa7a9228-9f50-4a01-8cba-125681b964d7-kube-api-access-pkz89\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:57:38.999157 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:38.999134    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"]
Sep 10 23:57:39.001571 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:39.001543    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e1f72544/e2e-lifecycle-model-e1f72544-kserve-68dcd44875-lc96x"]
Sep 10 23:57:40.656118 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:40.656082    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7a9228-9f50-4a01-8cba-125681b964d7" path="/var/lib/kubelet/pods/aa7a9228-9f50-4a01-8cba-125681b964d7/volumes"
Sep 10 23:57:40.932337 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:40.932307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:41.566480 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:41.566450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:47.123411 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:47.123381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:47.194433 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:47.194406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:47.396540 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:47.396461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:48.616054 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:48.616013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:48.704842 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:48.704813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:48.804174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:48.804146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:49.391779 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:49.391722    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:49.492178 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:49.492148    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:49.589590 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:49.589562    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:50.758301 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:50.758266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:56.635282 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:56.635246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:56.696145 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:56.696112    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:57:57.322183 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:57:57.322150    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:03.220745 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:03.220707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:03.326857 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:03.326825    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:03.419395 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:03.419358    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:13.882251 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:13.882215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:14.100315 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:14.100281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:33.500917 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.500881    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7692-ad8f10"]
Sep 10 23:58:33.501284 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.501274    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5795bc78-ac04-48f7-9302-139e84fb2915" containerName="maas-api"
Sep 10 23:58:33.501327 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.501287    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5795bc78-ac04-48f7-9302-139e84fb2915" containerName="maas-api"
Sep 10 23:58:33.501327 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.501302    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7a9228-9f50-4a01-8cba-125681b964d7" containerName="main"
Sep 10 23:58:33.501327 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.501307    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7a9228-9f50-4a01-8cba-125681b964d7" containerName="main"
Sep 10 23:58:33.501423 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.501382    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7a9228-9f50-4a01-8cba-125681b964d7" containerName="main"
Sep 10 23:58:33.501423 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.501404    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5795bc78-ac04-48f7-9302-139e84fb2915" containerName="maas-api"
Sep 10 23:58:33.505465 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.505444    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-ad8f10"
Sep 10 23:58:33.507748 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.507723    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nfkgj\""
Sep 10 23:58:33.510625 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.510602    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-ad8f10"]
Sep 10 23:58:33.584469 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.584443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btbv\" (UniqueName: \"kubernetes.io/projected/63adbcd1-b412-470d-8f2f-64765e440185-kube-api-access-5btbv\") pod \"test-curl-7692-ad8f10\" (UID: \"63adbcd1-b412-470d-8f2f-64765e440185\") " pod="opendatahub/test-curl-7692-ad8f10"
Sep 10 23:58:33.685333 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.685299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btbv\" (UniqueName: \"kubernetes.io/projected/63adbcd1-b412-470d-8f2f-64765e440185-kube-api-access-5btbv\") pod \"test-curl-7692-ad8f10\" (UID: \"63adbcd1-b412-470d-8f2f-64765e440185\") " pod="opendatahub/test-curl-7692-ad8f10"
Sep 10 23:58:33.693091 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.693062    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btbv\" (UniqueName: \"kubernetes.io/projected/63adbcd1-b412-470d-8f2f-64765e440185-kube-api-access-5btbv\") pod \"test-curl-7692-ad8f10\" (UID: \"63adbcd1-b412-470d-8f2f-64765e440185\") " pod="opendatahub/test-curl-7692-ad8f10"
Sep 10 23:58:33.816435 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:33.816355    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-ad8f10"
Sep 10 23:58:34.084065 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:34.084043    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-ad8f10"]
Sep 10 23:58:34.086180 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:58:34.086147    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63adbcd1_b412_470d_8f2f_64765e440185.slice/crio-ec64cdfa9eba0cec5ac1bf6bcc6c42a26f1739fd607cab787960652dd1a7a803 WatchSource:0}: Error finding container ec64cdfa9eba0cec5ac1bf6bcc6c42a26f1739fd607cab787960652dd1a7a803: Status 404 returned error can't find the container with id ec64cdfa9eba0cec5ac1bf6bcc6c42a26f1739fd607cab787960652dd1a7a803
Sep 10 23:58:34.949092 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:34.949056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-ad8f10" event={"ID":"63adbcd1-b412-470d-8f2f-64765e440185","Type":"ContainerStarted","Data":"d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b"}
Sep 10 23:58:34.949471 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:34.949102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-ad8f10" event={"ID":"63adbcd1-b412-470d-8f2f-64765e440185","Type":"ContainerStarted","Data":"ec64cdfa9eba0cec5ac1bf6bcc6c42a26f1739fd607cab787960652dd1a7a803"}
Sep 10 23:58:34.963483 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:34.963441    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7692-ad8f10" podStartSLOduration=1.963426058 podStartE2EDuration="1.963426058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:58:34.962632637 +0000 UTC m=+1326.793660172" watchObservedRunningTime="2026-09-10 23:58:34.963426058 +0000 UTC m=+1326.794453591"
Sep 10 23:58:35.000947 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:35.000910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:35.539760 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:35.539727    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7692-ad8f10"]
Sep 10 23:58:35.546081 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:35.546055    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7692-ad8f10"]
Sep 10 23:58:35.953899 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:58:35.953872    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7692-ad8f10" container="test-curl-7692-ad8f10"
Sep 10 23:58:36.916952 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:36.916906    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"]
Sep 10 23:58:36.938281 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:36.938255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:36.958180 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:36.958129    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7692-ad8f10" podUID="63adbcd1-b412-470d-8f2f-64765e440185" containerName="test-curl-7692-ad8f10" containerID="cri-o://d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b" gracePeriod=2
Sep 10 23:58:37.934153 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:37.934121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:39.130212 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.130189    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-ad8f10"
Sep 10 23:58:39.231381 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.231316    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5btbv\" (UniqueName: \"kubernetes.io/projected/63adbcd1-b412-470d-8f2f-64765e440185-kube-api-access-5btbv\") pod \"63adbcd1-b412-470d-8f2f-64765e440185\" (UID: \"63adbcd1-b412-470d-8f2f-64765e440185\") "
Sep 10 23:58:39.233277 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.233247    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63adbcd1-b412-470d-8f2f-64765e440185-kube-api-access-5btbv" (OuterVolumeSpecName: "kube-api-access-5btbv") pod "63adbcd1-b412-470d-8f2f-64765e440185" (UID: "63adbcd1-b412-470d-8f2f-64765e440185"). InnerVolumeSpecName "kube-api-access-5btbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:58:39.332267 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.332244    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5btbv\" (UniqueName: \"kubernetes.io/projected/63adbcd1-b412-470d-8f2f-64765e440185-kube-api-access-5btbv\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:58:39.972325 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.972295    2562 generic.go:356] "Generic (PLEG): container finished" podID="63adbcd1-b412-470d-8f2f-64765e440185" containerID="d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b" exitCode=137
Sep 10 23:58:39.972497 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.972336    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-ad8f10"
Sep 10 23:58:39.972497 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.972393    2562 scope.go:117] "RemoveContainer" containerID="d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b"
Sep 10 23:58:39.982277 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.982260    2562 scope.go:117] "RemoveContainer" containerID="d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b"
Sep 10 23:58:39.982514 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:58:39.982497    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b\": container with ID starting with d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b not found: ID does not exist" containerID="d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b"
Sep 10 23:58:39.982603 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:39.982520    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b"} err="failed to get container status \"d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b\": rpc error: code = NotFound desc = could not find container \"d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b\": container with ID starting with d1842fb009e924811995fdf724c08df8a5f607a5a7d8bd3a7c52d1dd809ce59b not found: ID does not exist"
Sep 10 23:58:39.983993 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:58:39.983976    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-ad8f10\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="63adbcd1-b412-470d-8f2f-64765e440185" pod="opendatahub/test-curl-7692-ad8f10"
Sep 10 23:58:40.655770 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:40.655736    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63adbcd1-b412-470d-8f2f-64765e440185" path="/var/lib/kubelet/pods/63adbcd1-b412-470d-8f2f-64765e440185/volumes"
Sep 10 23:58:40.830608 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:40.830578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:40.926174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:40.926101    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:41.166271 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:41.166240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:41.466411 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:41.466376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:41.641405 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:41.641376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:44.023356 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:44.023325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:44.123100 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:44.123063    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:45.190796 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.190765    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"]
Sep 10 23:58:45.191381 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.191360    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63adbcd1-b412-470d-8f2f-64765e440185" containerName="test-curl-7692-ad8f10"
Sep 10 23:58:45.191429 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.191385    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="63adbcd1-b412-470d-8f2f-64765e440185" containerName="test-curl-7692-ad8f10"
Sep 10 23:58:45.191464 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.191457    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="63adbcd1-b412-470d-8f2f-64765e440185" containerName="test-curl-7692-ad8f10"
Sep 10 23:58:45.195997 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.195971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.198650 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.198633    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8b47cf-serving-cert\""
Sep 10 23:58:45.205690 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.205666    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"]
Sep 10 23:58:45.228807 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.228786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:45.281071 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.281034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60936465-4f33-4baa-a4f3-42d8294c8e55-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.281199 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.281151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.281281 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.281259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-metrics-certs\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.281344 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.281312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58w6r\" (UniqueName: \"kubernetes.io/projected/60936465-4f33-4baa-a4f3-42d8294c8e55-kube-api-access-58w6r\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.299721 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.299693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:45.382191 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.382164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60936465-4f33-4baa-a4f3-42d8294c8e55-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.382292 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.382200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.382292 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.382217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-metrics-certs\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.382292 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.382244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58w6r\" (UniqueName: \"kubernetes.io/projected/60936465-4f33-4baa-a4f3-42d8294c8e55-kube-api-access-58w6r\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.382461 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:58:45.382402    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-serving-cert: secret "maas-api-e2e-api-a-8b47cf-serving-cert" not found
Sep 10 23:58:45.382516 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:58:45.382486    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls podName:60936465-4f33-4baa-a4f3-42d8294c8e55 nodeName:}" failed. No retries permitted until 2026-09-10 23:58:45.882465508 +0000 UTC m=+1337.713493033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls") pod "maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp" (UID: "60936465-4f33-4baa-a4f3-42d8294c8e55") : secret "maas-api-e2e-api-a-8b47cf-serving-cert" not found
Sep 10 23:58:45.382792 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.382773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60936465-4f33-4baa-a4f3-42d8294c8e55-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.384547 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.384526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-metrics-certs\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.391174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.391155    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58w6r\" (UniqueName: \"kubernetes.io/projected/60936465-4f33-4baa-a4f3-42d8294c8e55-kube-api-access-58w6r\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.557856 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.557786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:45.712838 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.712805    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:45.815457 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.815380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:58:45.824469 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.824445    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"]
Sep 10 23:58:45.828261 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.828242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:45.830642 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.830625    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8b47cf-dockercfg-8hkp6\""
Sep 10 23:58:45.830973 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.830956    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8b47cf\""
Sep 10 23:58:45.842540 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.842517    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"]
Sep 10 23:58:45.862839 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.862814    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"]
Sep 10 23:58:45.866348 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.866328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:45.876690 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.876669    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"]
Sep 10 23:58:45.886960 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.886919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.887048 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.886997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t\" (UID: \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:45.887108 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.887072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg64h\" (UniqueName: \"kubernetes.io/projected/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-kube-api-access-bg64h\") pod \"payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t\" (UID: \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:45.889181 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.889164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls\") pod \"maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:45.987962 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.987917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg64h\" (UniqueName: \"kubernetes.io/projected/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-kube-api-access-bg64h\") pod \"payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t\" (UID: \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:45.988095 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.987998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ns2f\" (UniqueName: \"kubernetes.io/projected/8702a48f-9726-4dc3-a113-b5e25de94ac3-kube-api-access-2ns2f\") pod \"payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc\" (UID: \"8702a48f-9726-4dc3-a113-b5e25de94ac3\") " pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:45.988095 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.988078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t\" (UID: \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:45.988172 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.988129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8702a48f-9726-4dc3-a113-b5e25de94ac3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc\" (UID: \"8702a48f-9726-4dc3-a113-b5e25de94ac3\") " pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:45.988683 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.988664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t\" (UID: \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:45.995498 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:45.995473    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg64h\" (UniqueName: \"kubernetes.io/projected/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-kube-api-access-bg64h\") pod \"payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t\" (UID: \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:46.088550 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.088492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ns2f\" (UniqueName: \"kubernetes.io/projected/8702a48f-9726-4dc3-a113-b5e25de94ac3-kube-api-access-2ns2f\") pod \"payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc\" (UID: \"8702a48f-9726-4dc3-a113-b5e25de94ac3\") " pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:46.088650 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.088550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8702a48f-9726-4dc3-a113-b5e25de94ac3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc\" (UID: \"8702a48f-9726-4dc3-a113-b5e25de94ac3\") " pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:46.089228 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.089209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8702a48f-9726-4dc3-a113-b5e25de94ac3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc\" (UID: \"8702a48f-9726-4dc3-a113-b5e25de94ac3\") " pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:46.096310 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.096287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ns2f\" (UniqueName: \"kubernetes.io/projected/8702a48f-9726-4dc3-a113-b5e25de94ac3-kube-api-access-2ns2f\") pod \"payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc\" (UID: \"8702a48f-9726-4dc3-a113-b5e25de94ac3\") " pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:46.107498 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.107481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:46.138530 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.138508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:46.176592 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.176510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:46.381446 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.381418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"]
Sep 10 23:58:46.383259 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:58:46.383231    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60936465_4f33_4baa_a4f3_42d8294c8e55.slice/crio-565ac13c0711391301d5130718de8a922231ebaaab72a6e2a911193973ea30c2 WatchSource:0}: Error finding container 565ac13c0711391301d5130718de8a922231ebaaab72a6e2a911193973ea30c2: Status 404 returned error can't find the container with id 565ac13c0711391301d5130718de8a922231ebaaab72a6e2a911193973ea30c2
Sep 10 23:58:46.384902 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.384881    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:58:46.422504 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.422476    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"]
Sep 10 23:58:46.426357 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:58:46.426326    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e9f4ac_09bc_4c07_9b57_332ca5afdceb.slice/crio-0a0e29a624203a30ed51dba71836872b84443ac552c0a856ad65112dac0abf3f WatchSource:0}: Error finding container 0a0e29a624203a30ed51dba71836872b84443ac552c0a856ad65112dac0abf3f: Status 404 returned error can't find the container with id 0a0e29a624203a30ed51dba71836872b84443ac552c0a856ad65112dac0abf3f
Sep 10 23:58:46.462153 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:46.462055    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"]
Sep 10 23:58:46.474041 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:58:46.474010    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8702a48f_9726_4dc3_a113_b5e25de94ac3.slice/crio-bd0fb023359cda534fb99e0a0183e5c6dbda9cf3effcad75f04d29f4075aabf6 WatchSource:0}: Error finding container bd0fb023359cda534fb99e0a0183e5c6dbda9cf3effcad75f04d29f4075aabf6: Status 404 returned error can't find the container with id bd0fb023359cda534fb99e0a0183e5c6dbda9cf3effcad75f04d29f4075aabf6
Sep 10 23:58:47.010949 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.010835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc" event={"ID":"8702a48f-9726-4dc3-a113-b5e25de94ac3","Type":"ContainerStarted","Data":"b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3"}
Sep 10 23:58:47.010949 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.010903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc" event={"ID":"8702a48f-9726-4dc3-a113-b5e25de94ac3","Type":"ContainerStarted","Data":"bd0fb023359cda534fb99e0a0183e5c6dbda9cf3effcad75f04d29f4075aabf6"}
Sep 10 23:58:47.010949 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.010949    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:58:47.012455 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.012432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp" event={"ID":"60936465-4f33-4baa-a4f3-42d8294c8e55","Type":"ContainerStarted","Data":"cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f"}
Sep 10 23:58:47.012554 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.012473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp" event={"ID":"60936465-4f33-4baa-a4f3-42d8294c8e55","Type":"ContainerStarted","Data":"565ac13c0711391301d5130718de8a922231ebaaab72a6e2a911193973ea30c2"}
Sep 10 23:58:47.012554 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.012498    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:47.013972 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.013949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t" event={"ID":"91e9f4ac-09bc-4c07-9b57-332ca5afdceb","Type":"ContainerStarted","Data":"65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408"}
Sep 10 23:58:47.014065 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.014018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t" event={"ID":"91e9f4ac-09bc-4c07-9b57-332ca5afdceb","Type":"ContainerStarted","Data":"0a0e29a624203a30ed51dba71836872b84443ac552c0a856ad65112dac0abf3f"}
Sep 10 23:58:47.014065 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.014044    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:47.030943 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.030890    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc" podStartSLOduration=2.030876298 podStartE2EDuration="2.030876298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:58:47.028587826 +0000 UTC m=+1338.859615360" watchObservedRunningTime="2026-09-10 23:58:47.030876298 +0000 UTC m=+1338.861903832"
Sep 10 23:58:47.046914 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.046875    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp" podStartSLOduration=1.680020372 podStartE2EDuration="2.046862354s" totalImagesPullingTime="366.841982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:58:47.046030253 +0000 UTC m=+1338.877057807" watchObservedRunningTime="2026-09-10 23:58:47.046862354 +0000 UTC m=+1338.877889886"
Sep 10 23:58:47.063507 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:47.063467    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t" podStartSLOduration=2.063454805 podStartE2EDuration="2.063454805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:58:47.06137148 +0000 UTC m=+1338.892399035" watchObservedRunningTime="2026-09-10 23:58:47.063454805 +0000 UTC m=+1338.894482375"
Sep 10 23:58:51.936416 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:51.936329    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm" podUID="4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" containerName="main" containerID="cri-o://18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5" gracePeriod=45
Sep 10 23:58:52.191734 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.191678    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:58:52.347009 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.346976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tls-certs\") pod \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") "
Sep 10 23:58:52.347183 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347028    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tmp-dir\") pod \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") "
Sep 10 23:58:52.347183 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347117    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-home\") pod \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") "
Sep 10 23:58:52.347183 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347161    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-model-cache\") pod \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") "
Sep 10 23:58:52.347346 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347198    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-dshm\") pod \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") "
Sep 10 23:58:52.347346 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347230    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j489t\" (UniqueName: \"kubernetes.io/projected/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-kube-api-access-j489t\") pod \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\" (UID: \"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e\") "
Sep 10 23:58:52.347460 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347358    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" (UID: "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:52.347460 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347371    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-home" (OuterVolumeSpecName: "home") pod "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" (UID: "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:52.347460 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347448    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" (UID: "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:52.347605 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347588    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:58:52.347660 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347607    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:58:52.347660 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.347617    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:58:52.349327 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.349293    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" (UID: "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:58:52.349327 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.349293    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-dshm" (OuterVolumeSpecName: "dshm") pod "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" (UID: "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:52.349506 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.349380    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-kube-api-access-j489t" (OuterVolumeSpecName: "kube-api-access-j489t") pod "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" (UID: "4ffd6746-3b05-4aeb-9bd4-9af8ed84892e"). InnerVolumeSpecName "kube-api-access-j489t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:58:52.448661 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.448576    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:58:52.448661 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.448612    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j489t\" (UniqueName: \"kubernetes.io/projected/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-kube-api-access-j489t\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:58:52.448661 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:52.448624    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:58:53.025823 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.025800    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:58:53.047887 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.047858    2562 generic.go:356] "Generic (PLEG): container finished" podID="4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" containerID="18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5" exitCode=0
Sep 10 23:58:53.048031 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.047893    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"
Sep 10 23:58:53.048031 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.047964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm" event={"ID":"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e","Type":"ContainerDied","Data":"18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5"}
Sep 10 23:58:53.048031 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.048001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm" event={"ID":"4ffd6746-3b05-4aeb-9bd4-9af8ed84892e","Type":"ContainerDied","Data":"d00d41014bf71d2bcf35a5a34ba257cfa79547a12f1b9717fe780f42a8e45523"}
Sep 10 23:58:53.048031 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.048020    2562 scope.go:117] "RemoveContainer" containerID="18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5"
Sep 10 23:58:53.057668 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.057647    2562 scope.go:117] "RemoveContainer" containerID="18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5"
Sep 10 23:58:53.057995 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:58:53.057973    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5\": container with ID starting with 18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5 not found: ID does not exist" containerID="18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5"
Sep 10 23:58:53.058102 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.058005    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5"} err="failed to get container status \"18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5\": rpc error: code = NotFound desc = could not find container \"18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5\": container with ID starting with 18fefc2f6763a38ef9e486ffb0b9a123e7ab99a3721e4d2dacfbc14726d11ae5 not found: ID does not exist"
Sep 10 23:58:53.066233 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.066212    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"]
Sep 10 23:58:53.070094 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:53.070072    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/auth-test-model-d7a539-kserve-7995686c99-bcmbm"]
Sep 10 23:58:54.661511 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:54.661476    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" path="/var/lib/kubelet/pods/4ffd6746-3b05-4aeb-9bd4-9af8ed84892e/volumes"
Sep 10 23:58:58.023001 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:58.022971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:58:58.023422 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:58:58.023036    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:59:03.770156 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:03.770111    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:08.981188 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:08.981157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:09.422716 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:09.422635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:09.729542 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:09.729483    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:10.240696 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:10.240657    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:13.235296 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:13.235259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:13.325498 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:13.325466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:14.342145 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.342114    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"]
Sep 10 23:59:14.343060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.342747    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" containerName="main"
Sep 10 23:59:14.343060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.342771    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" containerName="main"
Sep 10 23:59:14.343060 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.342941    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffd6746-3b05-4aeb-9bd4-9af8ed84892e" containerName="main"
Sep 10 23:59:14.346357 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.346337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.348713 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.348689    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4ec4c8-serving-cert\""
Sep 10 23:59:14.358556 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.358534    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"]
Sep 10 23:59:14.381481 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.381455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:14.430382 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.430343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-maas-api-tls\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.430487 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.430389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-metrics-certs\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.430487 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.430424    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xmn5\" (UniqueName: \"kubernetes.io/projected/e830395e-9825-44e2-bd97-0c76ce99770e-kube-api-access-9xmn5\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.430487 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.430466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830395e-9825-44e2-bd97-0c76ce99770e-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.488230 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.488207    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:14.531723 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.531704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830395e-9825-44e2-bd97-0c76ce99770e-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.531802 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.531767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-maas-api-tls\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.531802 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.531795    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-metrics-certs\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.531892 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.531824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmn5\" (UniqueName: \"kubernetes.io/projected/e830395e-9825-44e2-bd97-0c76ce99770e-kube-api-access-9xmn5\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.532462 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.532437    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830395e-9825-44e2-bd97-0c76ce99770e-openshift-service-ca\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.534360 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.534329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-maas-api-tls\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.534460 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.534328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-metrics-certs\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.539485 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.539463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmn5\" (UniqueName: \"kubernetes.io/projected/e830395e-9825-44e2-bd97-0c76ce99770e-kube-api-access-9xmn5\") pod \"maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.581438 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.581414    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:14.658037 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.657973    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:14.867729 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.867699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:14.929362 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.929333    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"]
Sep 10 23:59:14.930981 ip-10-0-137-218 kubenswrapper[2562]: W0910 23:59:14.930954    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode830395e_9825_44e2_bd97_0c76ce99770e.slice/crio-cbe5200b99b5af02be007bf1cef2061f0908360b6e61601d15bb0cc0aa9f5506 WatchSource:0}: Error finding container cbe5200b99b5af02be007bf1cef2061f0908360b6e61601d15bb0cc0aa9f5506: Status 404 returned error can't find the container with id cbe5200b99b5af02be007bf1cef2061f0908360b6e61601d15bb0cc0aa9f5506
Sep 10 23:59:14.936186 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:14.936148    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:15.159409 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:15.159374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c" event={"ID":"e830395e-9825-44e2-bd97-0c76ce99770e","Type":"ContainerStarted","Data":"cbe5200b99b5af02be007bf1cef2061f0908360b6e61601d15bb0cc0aa9f5506"}
Sep 10 23:59:16.167412 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:16.167380    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c" event={"ID":"e830395e-9825-44e2-bd97-0c76ce99770e","Type":"ContainerStarted","Data":"343cdd88950b3f515eefb8ab7c64e57f5fbd3416fb27b19d59f35cfff3eb362d"}
Sep 10 23:59:16.167860 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:16.167500    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:16.186615 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:16.186560    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c" podStartSLOduration=1.695877444 podStartE2EDuration="2.186547753s" totalImagesPullingTime="490.670309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:16.184891874 +0000 UTC m=+1368.015919432" watchObservedRunningTime="2026-09-10 23:59:16.186547753 +0000 UTC m=+1368.017575286"
Sep 10 23:59:22.177691 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:22.177662    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 10 23:59:37.262657 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:37.262622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:38.233045 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:38.233000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:41.220493 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:41.220443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:56.543019 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.542984    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"]
Sep 10 23:59:56.543663 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.543273    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp" podUID="60936465-4f33-4baa-a4f3-42d8294c8e55" containerName="maas-api" containerID="cri-o://cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f" gracePeriod=30
Sep 10 23:59:56.576579 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.576553    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:56.595383 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.595291    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"]
Sep 10 23:59:56.595558 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.595526    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc" podUID="8702a48f-9726-4dc3-a113-b5e25de94ac3" containerName="payload-processing" containerID="cri-o://b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3" gracePeriod=30
Sep 10 23:59:56.602324 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.602299    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"]
Sep 10 23:59:56.602535 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.602495    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t" podUID="91e9f4ac-09bc-4c07-9b57-332ca5afdceb" containerName="payload-pre-processing" containerID="cri-o://65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408" gracePeriod=30
Sep 10 23:59:56.673481 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.673450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:56.761824 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.761792    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:56.822961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.822905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:56.891314 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.891292    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:59:56.893292 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.893277    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:59:56.906674 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.906654    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:59:56.984214 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984189    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58w6r\" (UniqueName: \"kubernetes.io/projected/60936465-4f33-4baa-a4f3-42d8294c8e55-kube-api-access-58w6r\") pod \"60936465-4f33-4baa-a4f3-42d8294c8e55\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") "
Sep 10 23:59:56.984396 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984225    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg64h\" (UniqueName: \"kubernetes.io/projected/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-kube-api-access-bg64h\") pod \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\" (UID: \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\") "
Sep 10 23:59:56.984396 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984245    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60936465-4f33-4baa-a4f3-42d8294c8e55-openshift-service-ca\") pod \"60936465-4f33-4baa-a4f3-42d8294c8e55\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") "
Sep 10 23:59:56.984396 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984269    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls\") pod \"60936465-4f33-4baa-a4f3-42d8294c8e55\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") "
Sep 10 23:59:56.984396 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984372    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ns2f\" (UniqueName: \"kubernetes.io/projected/8702a48f-9726-4dc3-a113-b5e25de94ac3-kube-api-access-2ns2f\") pod \"8702a48f-9726-4dc3-a113-b5e25de94ac3\" (UID: \"8702a48f-9726-4dc3-a113-b5e25de94ac3\") "
Sep 10 23:59:56.984620 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984477    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-plugins-config-volume\") pod \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\" (UID: \"91e9f4ac-09bc-4c07-9b57-332ca5afdceb\") "
Sep 10 23:59:56.984620 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984515    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-metrics-certs\") pod \"60936465-4f33-4baa-a4f3-42d8294c8e55\" (UID: \"60936465-4f33-4baa-a4f3-42d8294c8e55\") "
Sep 10 23:59:56.984620 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984561    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8702a48f-9726-4dc3-a113-b5e25de94ac3-plugins-config-volume\") pod \"8702a48f-9726-4dc3-a113-b5e25de94ac3\" (UID: \"8702a48f-9726-4dc3-a113-b5e25de94ac3\") "
Sep 10 23:59:56.984785 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984610    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60936465-4f33-4baa-a4f3-42d8294c8e55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60936465-4f33-4baa-a4f3-42d8294c8e55" (UID: "60936465-4f33-4baa-a4f3-42d8294c8e55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:59:56.984941 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.984904    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60936465-4f33-4baa-a4f3-42d8294c8e55-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:59:56.985033 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.985004    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91e9f4ac-09bc-4c07-9b57-332ca5afdceb" (UID: "91e9f4ac-09bc-4c07-9b57-332ca5afdceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:59:56.985299 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.985273    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8702a48f-9726-4dc3-a113-b5e25de94ac3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8702a48f-9726-4dc3-a113-b5e25de94ac3" (UID: "8702a48f-9726-4dc3-a113-b5e25de94ac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:59:56.986790 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.986767    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-kube-api-access-bg64h" (OuterVolumeSpecName: "kube-api-access-bg64h") pod "91e9f4ac-09bc-4c07-9b57-332ca5afdceb" (UID: "91e9f4ac-09bc-4c07-9b57-332ca5afdceb"). InnerVolumeSpecName "kube-api-access-bg64h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:59:56.986961 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.986914    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60936465-4f33-4baa-a4f3-42d8294c8e55" (UID: "60936465-4f33-4baa-a4f3-42d8294c8e55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:59:56.987044 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.986997    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "60936465-4f33-4baa-a4f3-42d8294c8e55" (UID: "60936465-4f33-4baa-a4f3-42d8294c8e55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:59:56.987044 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.987020    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60936465-4f33-4baa-a4f3-42d8294c8e55-kube-api-access-58w6r" (OuterVolumeSpecName: "kube-api-access-58w6r") pod "60936465-4f33-4baa-a4f3-42d8294c8e55" (UID: "60936465-4f33-4baa-a4f3-42d8294c8e55"). InnerVolumeSpecName "kube-api-access-58w6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:59:56.987129 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:56.987080    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8702a48f-9726-4dc3-a113-b5e25de94ac3-kube-api-access-2ns2f" (OuterVolumeSpecName: "kube-api-access-2ns2f") pod "8702a48f-9726-4dc3-a113-b5e25de94ac3" (UID: "8702a48f-9726-4dc3-a113-b5e25de94ac3"). InnerVolumeSpecName "kube-api-access-2ns2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:59:57.085755 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.085694    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ns2f\" (UniqueName: \"kubernetes.io/projected/8702a48f-9726-4dc3-a113-b5e25de94ac3-kube-api-access-2ns2f\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:59:57.085755 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.085716    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:59:57.085755 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.085726    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:59:57.085755 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.085735    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8702a48f-9726-4dc3-a113-b5e25de94ac3-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:59:57.085755 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.085744    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58w6r\" (UniqueName: \"kubernetes.io/projected/60936465-4f33-4baa-a4f3-42d8294c8e55-kube-api-access-58w6r\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:59:57.085755 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.085752    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg64h\" (UniqueName: \"kubernetes.io/projected/91e9f4ac-09bc-4c07-9b57-332ca5afdceb-kube-api-access-bg64h\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:59:57.086051 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.085760    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60936465-4f33-4baa-a4f3-42d8294c8e55-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 10 23:59:57.106608 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.106578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 10 23:59:57.370513 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.370436    2562 generic.go:356] "Generic (PLEG): container finished" podID="60936465-4f33-4baa-a4f3-42d8294c8e55" containerID="cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f" exitCode=0
Sep 10 23:59:57.370513 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.370479    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"
Sep 10 23:59:57.370719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.370521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp" event={"ID":"60936465-4f33-4baa-a4f3-42d8294c8e55","Type":"ContainerDied","Data":"cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f"}
Sep 10 23:59:57.370719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.370571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp" event={"ID":"60936465-4f33-4baa-a4f3-42d8294c8e55","Type":"ContainerDied","Data":"565ac13c0711391301d5130718de8a922231ebaaab72a6e2a911193973ea30c2"}
Sep 10 23:59:57.370719 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.370594    2562 scope.go:117] "RemoveContainer" containerID="cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f"
Sep 10 23:59:57.372117 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.372092    2562 generic.go:356] "Generic (PLEG): container finished" podID="8702a48f-9726-4dc3-a113-b5e25de94ac3" containerID="b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3" exitCode=0
Sep 10 23:59:57.372235 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.372120    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc" event={"ID":"8702a48f-9726-4dc3-a113-b5e25de94ac3","Type":"ContainerDied","Data":"b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3"}
Sep 10 23:59:57.372235 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.372153    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"
Sep 10 23:59:57.372235 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.372167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc" event={"ID":"8702a48f-9726-4dc3-a113-b5e25de94ac3","Type":"ContainerDied","Data":"bd0fb023359cda534fb99e0a0183e5c6dbda9cf3effcad75f04d29f4075aabf6"}
Sep 10 23:59:57.373686 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.373661    2562 generic.go:356] "Generic (PLEG): container finished" podID="91e9f4ac-09bc-4c07-9b57-332ca5afdceb" containerID="65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408" exitCode=0
Sep 10 23:59:57.373796 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.373694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t" event={"ID":"91e9f4ac-09bc-4c07-9b57-332ca5afdceb","Type":"ContainerDied","Data":"65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408"}
Sep 10 23:59:57.373796 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.373700    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"
Sep 10 23:59:57.373796 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.373725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t" event={"ID":"91e9f4ac-09bc-4c07-9b57-332ca5afdceb","Type":"ContainerDied","Data":"0a0e29a624203a30ed51dba71836872b84443ac552c0a856ad65112dac0abf3f"}
Sep 10 23:59:57.380695 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.380519    2562 scope.go:117] "RemoveContainer" containerID="cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f"
Sep 10 23:59:57.380911 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:59:57.380890    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f\": container with ID starting with cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f not found: ID does not exist" containerID="cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f"
Sep 10 23:59:57.381017 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.380918    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f"} err="failed to get container status \"cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f\": rpc error: code = NotFound desc = could not find container \"cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f\": container with ID starting with cd8aa86c774b98ecac1836eda827c4324b988d79dbb82e5480c6b68102e1cc0f not found: ID does not exist"
Sep 10 23:59:57.381017 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.380954    2562 scope.go:117] "RemoveContainer" containerID="b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3"
Sep 10 23:59:57.389737 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.389719    2562 scope.go:117] "RemoveContainer" containerID="b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3"
Sep 10 23:59:57.390012 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:59:57.389990    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3\": container with ID starting with b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3 not found: ID does not exist" containerID="b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3"
Sep 10 23:59:57.390077 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.390020    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3"} err="failed to get container status \"b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3\": rpc error: code = NotFound desc = could not find container \"b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3\": container with ID starting with b8c96722af568e1009b6231777633cecc0a62bddc276ef7c8830eec614c8e1b3 not found: ID does not exist"
Sep 10 23:59:57.390077 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.390039    2562 scope.go:117] "RemoveContainer" containerID="65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408"
Sep 10 23:59:57.397589 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.397572    2562 scope.go:117] "RemoveContainer" containerID="65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408"
Sep 10 23:59:57.397822 ip-10-0-137-218 kubenswrapper[2562]: E0910 23:59:57.397806    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408\": container with ID starting with 65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408 not found: ID does not exist" containerID="65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408"
Sep 10 23:59:57.397866 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.397827    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408"} err="failed to get container status \"65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408\": rpc error: code = NotFound desc = could not find container \"65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408\": container with ID starting with 65df7d8742c262d6399222c6a7b39fe6543dc1e6e6fd11b9bc31a8ab5e84b408 not found: ID does not exist"
Sep 10 23:59:57.416393 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.416369    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"]
Sep 10 23:59:57.426269 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.426250    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b47cf-f8497d9b5-tw6xp"]
Sep 10 23:59:57.478598 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.478579    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"]
Sep 10 23:59:57.492331 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.492308    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8b47cf-75476bc5cf-q6vwc"]
Sep 10 23:59:57.530170 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.530149    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"]
Sep 10 23:59:57.533489 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:57.533469    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8b47cf-cb6486458-hn74t"]
Sep 10 23:59:58.655667 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:58.655624    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60936465-4f33-4baa-a4f3-42d8294c8e55" path="/var/lib/kubelet/pods/60936465-4f33-4baa-a4f3-42d8294c8e55/volumes"
Sep 10 23:59:58.656174 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:58.656158    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8702a48f-9726-4dc3-a113-b5e25de94ac3" path="/var/lib/kubelet/pods/8702a48f-9726-4dc3-a113-b5e25de94ac3/volumes"
Sep 10 23:59:58.656539 ip-10-0-137-218 kubenswrapper[2562]: I0910 23:59:58.656517    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e9f4ac-09bc-4c07-9b57-332ca5afdceb" path="/var/lib/kubelet/pods/91e9f4ac-09bc-4c07-9b57-332ca5afdceb/volumes"
Sep 11 00:00:00.220712 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.220675    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29818080-ksv5c"]
Sep 11 00:00:00.221359 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221335    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8702a48f-9726-4dc3-a113-b5e25de94ac3" containerName="payload-processing"
Sep 11 00:00:00.221473 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221363    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8702a48f-9726-4dc3-a113-b5e25de94ac3" containerName="payload-processing"
Sep 11 00:00:00.221473 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221391    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e9f4ac-09bc-4c07-9b57-332ca5afdceb" containerName="payload-pre-processing"
Sep 11 00:00:00.221473 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221400    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e9f4ac-09bc-4c07-9b57-332ca5afdceb" containerName="payload-pre-processing"
Sep 11 00:00:00.221473 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221413    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60936465-4f33-4baa-a4f3-42d8294c8e55" containerName="maas-api"
Sep 11 00:00:00.221473 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221422    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60936465-4f33-4baa-a4f3-42d8294c8e55" containerName="maas-api"
Sep 11 00:00:00.221714 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221528    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8702a48f-9726-4dc3-a113-b5e25de94ac3" containerName="payload-processing"
Sep 11 00:00:00.221714 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221544    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="60936465-4f33-4baa-a4f3-42d8294c8e55" containerName="maas-api"
Sep 11 00:00:00.221714 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.221553    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e9f4ac-09bc-4c07-9b57-332ca5afdceb" containerName="payload-pre-processing"
Sep 11 00:00:00.225360 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.225335    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:00.229166 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.229141    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Sep 11 00:00:00.229269 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.229154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-6t9cr\""
Sep 11 00:00:00.245165 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.245135    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29818080-ksv5c"]
Sep 11 00:00:00.311877 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.311846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vx7h\" (UniqueName: \"kubernetes.io/projected/733adc40-b33d-4910-bed7-fe7cd1bef11d-kube-api-access-2vx7h\") pod \"image-pruner-29818080-ksv5c\" (UID: \"733adc40-b33d-4910-bed7-fe7cd1bef11d\") " pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:00.312063 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.311896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733adc40-b33d-4910-bed7-fe7cd1bef11d-serviceca\") pod \"image-pruner-29818080-ksv5c\" (UID: \"733adc40-b33d-4910-bed7-fe7cd1bef11d\") " pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:00.413207 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.413175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733adc40-b33d-4910-bed7-fe7cd1bef11d-serviceca\") pod \"image-pruner-29818080-ksv5c\" (UID: \"733adc40-b33d-4910-bed7-fe7cd1bef11d\") " pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:00.413344 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.413309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vx7h\" (UniqueName: \"kubernetes.io/projected/733adc40-b33d-4910-bed7-fe7cd1bef11d-kube-api-access-2vx7h\") pod \"image-pruner-29818080-ksv5c\" (UID: \"733adc40-b33d-4910-bed7-fe7cd1bef11d\") " pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:00.413744 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.413722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733adc40-b33d-4910-bed7-fe7cd1bef11d-serviceca\") pod \"image-pruner-29818080-ksv5c\" (UID: \"733adc40-b33d-4910-bed7-fe7cd1bef11d\") " pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:00.429738 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.429712    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vx7h\" (UniqueName: \"kubernetes.io/projected/733adc40-b33d-4910-bed7-fe7cd1bef11d-kube-api-access-2vx7h\") pod \"image-pruner-29818080-ksv5c\" (UID: \"733adc40-b33d-4910-bed7-fe7cd1bef11d\") " pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:00.555951 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.555862    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:00.824734 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:00.824659    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29818080-ksv5c"]
Sep 11 00:00:00.828029 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:00:00.828001    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733adc40_b33d_4910_bed7_fe7cd1bef11d.slice/crio-48e9e516271dc1974830e36fd5ce1fc197e74eb975b48b9ea7d1cc8f1d855523 WatchSource:0}: Error finding container 48e9e516271dc1974830e36fd5ce1fc197e74eb975b48b9ea7d1cc8f1d855523: Status 404 returned error can't find the container with id 48e9e516271dc1974830e36fd5ce1fc197e74eb975b48b9ea7d1cc8f1d855523
Sep 11 00:00:01.404450 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:01.404416    2562 generic.go:356] "Generic (PLEG): container finished" podID="733adc40-b33d-4910-bed7-fe7cd1bef11d" containerID="b64b85e37fdbc0156ed18a1785fed79e67c074d7bd602d147fc6996f2b544c1d" exitCode=0
Sep 11 00:00:01.404791 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:01.404507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29818080-ksv5c" event={"ID":"733adc40-b33d-4910-bed7-fe7cd1bef11d","Type":"ContainerDied","Data":"b64b85e37fdbc0156ed18a1785fed79e67c074d7bd602d147fc6996f2b544c1d"}
Sep 11 00:00:01.404791 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:01.404557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29818080-ksv5c" event={"ID":"733adc40-b33d-4910-bed7-fe7cd1bef11d","Type":"ContainerStarted","Data":"48e9e516271dc1974830e36fd5ce1fc197e74eb975b48b9ea7d1cc8f1d855523"}
Sep 11 00:00:02.553920 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:02.553897    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:02.737155 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:02.737127    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vx7h\" (UniqueName: \"kubernetes.io/projected/733adc40-b33d-4910-bed7-fe7cd1bef11d-kube-api-access-2vx7h\") pod \"733adc40-b33d-4910-bed7-fe7cd1bef11d\" (UID: \"733adc40-b33d-4910-bed7-fe7cd1bef11d\") "
Sep 11 00:00:02.737311 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:02.737184    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733adc40-b33d-4910-bed7-fe7cd1bef11d-serviceca\") pod \"733adc40-b33d-4910-bed7-fe7cd1bef11d\" (UID: \"733adc40-b33d-4910-bed7-fe7cd1bef11d\") "
Sep 11 00:00:02.739006 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:02.737748    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733adc40-b33d-4910-bed7-fe7cd1bef11d-serviceca" (OuterVolumeSpecName: "serviceca") pod "733adc40-b33d-4910-bed7-fe7cd1bef11d" (UID: "733adc40-b33d-4910-bed7-fe7cd1bef11d"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:02.739006 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:02.738053    2562 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733adc40-b33d-4910-bed7-fe7cd1bef11d-serviceca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:02.744339 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:02.740041    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733adc40-b33d-4910-bed7-fe7cd1bef11d-kube-api-access-2vx7h" (OuterVolumeSpecName: "kube-api-access-2vx7h") pod "733adc40-b33d-4910-bed7-fe7cd1bef11d" (UID: "733adc40-b33d-4910-bed7-fe7cd1bef11d"). InnerVolumeSpecName "kube-api-access-2vx7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:02.839451 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:02.839412    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vx7h\" (UniqueName: \"kubernetes.io/projected/733adc40-b33d-4910-bed7-fe7cd1bef11d-kube-api-access-2vx7h\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:03.416077 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:03.416044    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29818080-ksv5c"
Sep 11 00:00:03.416235 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:03.416081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29818080-ksv5c" event={"ID":"733adc40-b33d-4910-bed7-fe7cd1bef11d","Type":"ContainerDied","Data":"48e9e516271dc1974830e36fd5ce1fc197e74eb975b48b9ea7d1cc8f1d855523"}
Sep 11 00:00:03.416235 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:03.416162    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e9e516271dc1974830e36fd5ce1fc197e74eb975b48b9ea7d1cc8f1d855523"
Sep 11 00:00:03.505356 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:03.505325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:03.798592 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:03.798560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:04.388331 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:04.388298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:13.759821 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:13.759778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:19.280206 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.280174    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"]
Sep 11 00:00:19.280624 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.280374    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c" podUID="e830395e-9825-44e2-bd97-0c76ce99770e" containerName="maas-api" containerID="cri-o://343cdd88950b3f515eefb8ab7c64e57f5fbd3416fb27b19d59f35cfff3eb362d" gracePeriod=30
Sep 11 00:00:19.311536 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.311504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:19.410613 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.410581    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:19.497039 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.497002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:19.497938 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.497887    2562 generic.go:356] "Generic (PLEG): container finished" podID="e830395e-9825-44e2-bd97-0c76ce99770e" containerID="343cdd88950b3f515eefb8ab7c64e57f5fbd3416fb27b19d59f35cfff3eb362d" exitCode=0
Sep 11 00:00:19.498027 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.497965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c" event={"ID":"e830395e-9825-44e2-bd97-0c76ce99770e","Type":"ContainerDied","Data":"343cdd88950b3f515eefb8ab7c64e57f5fbd3416fb27b19d59f35cfff3eb362d"}
Sep 11 00:00:19.532213 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.532163    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 11 00:00:19.553282 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.553259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:19.583007 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.582981    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xmn5\" (UniqueName: \"kubernetes.io/projected/e830395e-9825-44e2-bd97-0c76ce99770e-kube-api-access-9xmn5\") pod \"e830395e-9825-44e2-bd97-0c76ce99770e\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") "
Sep 11 00:00:19.583129 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.583025    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830395e-9825-44e2-bd97-0c76ce99770e-openshift-service-ca\") pod \"e830395e-9825-44e2-bd97-0c76ce99770e\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") "
Sep 11 00:00:19.583129 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.583081    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-maas-api-tls\") pod \"e830395e-9825-44e2-bd97-0c76ce99770e\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") "
Sep 11 00:00:19.583244 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.583186    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-metrics-certs\") pod \"e830395e-9825-44e2-bd97-0c76ce99770e\" (UID: \"e830395e-9825-44e2-bd97-0c76ce99770e\") "
Sep 11 00:00:19.583499 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.583468    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e830395e-9825-44e2-bd97-0c76ce99770e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e830395e-9825-44e2-bd97-0c76ce99770e" (UID: "e830395e-9825-44e2-bd97-0c76ce99770e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:19.585225 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.585195    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e830395e-9825-44e2-bd97-0c76ce99770e-kube-api-access-9xmn5" (OuterVolumeSpecName: "kube-api-access-9xmn5") pod "e830395e-9825-44e2-bd97-0c76ce99770e" (UID: "e830395e-9825-44e2-bd97-0c76ce99770e"). InnerVolumeSpecName "kube-api-access-9xmn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:19.585225 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.585212    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e830395e-9825-44e2-bd97-0c76ce99770e" (UID: "e830395e-9825-44e2-bd97-0c76ce99770e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:00:19.585341 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.585226    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e830395e-9825-44e2-bd97-0c76ce99770e" (UID: "e830395e-9825-44e2-bd97-0c76ce99770e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:00:19.683855 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.683821    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:19.683855 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.683853    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xmn5\" (UniqueName: \"kubernetes.io/projected/e830395e-9825-44e2-bd97-0c76ce99770e-kube-api-access-9xmn5\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:19.684066 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.683866    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830395e-9825-44e2-bd97-0c76ce99770e-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:19.684066 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.683880    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830395e-9825-44e2-bd97-0c76ce99770e-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:19.847155 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:19.847071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:20.504346 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:20.504312    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"
Sep 11 00:00:20.504721 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:20.504347    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c" event={"ID":"e830395e-9825-44e2-bd97-0c76ce99770e","Type":"ContainerDied","Data":"cbe5200b99b5af02be007bf1cef2061f0908360b6e61601d15bb0cc0aa9f5506"}
Sep 11 00:00:20.504721 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:20.504403    2562 scope.go:117] "RemoveContainer" containerID="343cdd88950b3f515eefb8ab7c64e57f5fbd3416fb27b19d59f35cfff3eb362d"
Sep 11 00:00:20.525515 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:20.525491    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"]
Sep 11 00:00:20.528947 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:20.528913    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4ec4c8-5fd84b4c75-mdx5c"]
Sep 11 00:00:20.655799 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:20.655768    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e830395e-9825-44e2-bd97-0c76ce99770e" path="/var/lib/kubelet/pods/e830395e-9825-44e2-bd97-0c76ce99770e/volumes"
Sep 11 00:00:22.489530 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.489496    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7692-d490a5"]
Sep 11 00:00:22.489896 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.489879    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e830395e-9825-44e2-bd97-0c76ce99770e" containerName="maas-api"
Sep 11 00:00:22.489896 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.489890    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e830395e-9825-44e2-bd97-0c76ce99770e" containerName="maas-api"
Sep 11 00:00:22.490031 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.489914    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733adc40-b33d-4910-bed7-fe7cd1bef11d" containerName="image-pruner"
Sep 11 00:00:22.490031 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.489921    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="733adc40-b33d-4910-bed7-fe7cd1bef11d" containerName="image-pruner"
Sep 11 00:00:22.490031 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.489995    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="733adc40-b33d-4910-bed7-fe7cd1bef11d" containerName="image-pruner"
Sep 11 00:00:22.490031 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.490003    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e830395e-9825-44e2-bd97-0c76ce99770e" containerName="maas-api"
Sep 11 00:00:22.494431 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.494402    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:22.496707 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.496668    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-d490a5"]
Sep 11 00:00:22.496830 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.496802    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nfkgj\""
Sep 11 00:00:22.608069 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.608046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62v8k\" (UniqueName: \"kubernetes.io/projected/3ea20541-9f8c-4bee-9453-93312f12f0d3-kube-api-access-62v8k\") pod \"test-curl-7692-d490a5\" (UID: \"3ea20541-9f8c-4bee-9453-93312f12f0d3\") " pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:22.709042 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.709016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62v8k\" (UniqueName: \"kubernetes.io/projected/3ea20541-9f8c-4bee-9453-93312f12f0d3-kube-api-access-62v8k\") pod \"test-curl-7692-d490a5\" (UID: \"3ea20541-9f8c-4bee-9453-93312f12f0d3\") " pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:22.716800 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.716777    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62v8k\" (UniqueName: \"kubernetes.io/projected/3ea20541-9f8c-4bee-9453-93312f12f0d3-kube-api-access-62v8k\") pod \"test-curl-7692-d490a5\" (UID: \"3ea20541-9f8c-4bee-9453-93312f12f0d3\") " pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:22.804012 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:22.803943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:23.079148 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:23.079123    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-d490a5"]
Sep 11 00:00:23.081677 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:00:23.081643    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ea20541_9f8c_4bee_9453_93312f12f0d3.slice/crio-4b87dd3de436b642d512c287865bc900ffcea61a24e0f096363d84b55c9abe78 WatchSource:0}: Error finding container 4b87dd3de436b642d512c287865bc900ffcea61a24e0f096363d84b55c9abe78: Status 404 returned error can't find the container with id 4b87dd3de436b642d512c287865bc900ffcea61a24e0f096363d84b55c9abe78
Sep 11 00:00:23.522833 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:23.522796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-d490a5" event={"ID":"3ea20541-9f8c-4bee-9453-93312f12f0d3","Type":"ContainerStarted","Data":"cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5"}
Sep 11 00:00:23.522833 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:23.522837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-d490a5" event={"ID":"3ea20541-9f8c-4bee-9453-93312f12f0d3","Type":"ContainerStarted","Data":"4b87dd3de436b642d512c287865bc900ffcea61a24e0f096363d84b55c9abe78"}
Sep 11 00:00:23.538337 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:23.538292    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7692-d490a5" podStartSLOduration=1.538277915 podStartE2EDuration="1.538277915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:23.536290195 +0000 UTC m=+1435.367317728" watchObservedRunningTime="2026-09-11 00:00:23.538277915 +0000 UTC m=+1435.369305447"
Sep 11 00:00:24.107478 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.107444    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7692-d490a5"]
Sep 11 00:00:24.111738 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.111708    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7692-d490a5"]
Sep 11 00:00:24.220723 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.220693    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7692-61e3f2"]
Sep 11 00:00:24.221157 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.221137    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea20541-9f8c-4bee-9453-93312f12f0d3" containerName="test-curl-7692-d490a5"
Sep 11 00:00:24.221157 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.221158    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea20541-9f8c-4bee-9453-93312f12f0d3" containerName="test-curl-7692-d490a5"
Sep 11 00:00:24.221343 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.221262    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea20541-9f8c-4bee-9453-93312f12f0d3" containerName="test-curl-7692-d490a5"
Sep 11 00:00:24.224752 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.224732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-61e3f2"
Sep 11 00:00:24.229839 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.229817    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-61e3f2"]
Sep 11 00:00:24.321048 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.321015    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25j24\" (UniqueName: \"kubernetes.io/projected/61b02256-66a9-4854-ab3e-72df6ebb7ef7-kube-api-access-25j24\") pod \"test-curl-7692-61e3f2\" (UID: \"61b02256-66a9-4854-ab3e-72df6ebb7ef7\") " pod="opendatahub/test-curl-7692-61e3f2"
Sep 11 00:00:24.421650 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.421623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25j24\" (UniqueName: \"kubernetes.io/projected/61b02256-66a9-4854-ab3e-72df6ebb7ef7-kube-api-access-25j24\") pod \"test-curl-7692-61e3f2\" (UID: \"61b02256-66a9-4854-ab3e-72df6ebb7ef7\") " pod="opendatahub/test-curl-7692-61e3f2"
Sep 11 00:00:24.431199 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.431172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25j24\" (UniqueName: \"kubernetes.io/projected/61b02256-66a9-4854-ab3e-72df6ebb7ef7-kube-api-access-25j24\") pod \"test-curl-7692-61e3f2\" (UID: \"61b02256-66a9-4854-ab3e-72df6ebb7ef7\") " pod="opendatahub/test-curl-7692-61e3f2"
Sep 11 00:00:24.527342 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:24.527286    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7692-d490a5" container="test-curl-7692-d490a5"
Sep 11 00:00:24.535334 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.535315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-61e3f2"
Sep 11 00:00:24.803395 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:24.803370    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-61e3f2"]
Sep 11 00:00:24.805180 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:00:24.805148    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61b02256_66a9_4854_ab3e_72df6ebb7ef7.slice/crio-052b1aa5adc105c17c84281bb9e35f804093c5552f0f31d1b7d45ffb14d3a70a WatchSource:0}: Error finding container 052b1aa5adc105c17c84281bb9e35f804093c5552f0f31d1b7d45ffb14d3a70a: Status 404 returned error can't find the container with id 052b1aa5adc105c17c84281bb9e35f804093c5552f0f31d1b7d45ffb14d3a70a
Sep 11 00:00:25.533563 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:25.533522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-61e3f2" event={"ID":"61b02256-66a9-4854-ab3e-72df6ebb7ef7","Type":"ContainerStarted","Data":"cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3"}
Sep 11 00:00:25.533563 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:25.533553    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7692-d490a5" podUID="3ea20541-9f8c-4bee-9453-93312f12f0d3" containerName="test-curl-7692-d490a5" containerID="cri-o://cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5" gracePeriod=2
Sep 11 00:00:25.533563 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:25.533567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-61e3f2" event={"ID":"61b02256-66a9-4854-ab3e-72df6ebb7ef7","Type":"ContainerStarted","Data":"052b1aa5adc105c17c84281bb9e35f804093c5552f0f31d1b7d45ffb14d3a70a"}
Sep 11 00:00:25.555885 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:25.555842    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7692-61e3f2" podStartSLOduration=1.5558298289999999 podStartE2EDuration="1.555829829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:25.555121517 +0000 UTC m=+1437.386149037" watchObservedRunningTime="2026-09-11 00:00:25.555829829 +0000 UTC m=+1437.386857362"
Sep 11 00:00:25.889166 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:25.889092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:25.988198 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:25.988167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:26.154573 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:26.154488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:26.155460 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:26.155435    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7692-61e3f2"]
Sep 11 00:00:26.160508 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:26.160488    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7692-61e3f2"]
Sep 11 00:00:26.540086 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:26.540055    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7692-61e3f2" container="test-curl-7692-61e3f2"
Sep 11 00:00:27.551890 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:27.551853    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7692-61e3f2" podUID="61b02256-66a9-4854-ab3e-72df6ebb7ef7" containerName="test-curl-7692-61e3f2" containerID="cri-o://cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3" gracePeriod=2
Sep 11 00:00:27.715037 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:27.715010    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:27.853226 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:27.853163    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62v8k\" (UniqueName: \"kubernetes.io/projected/3ea20541-9f8c-4bee-9453-93312f12f0d3-kube-api-access-62v8k\") pod \"3ea20541-9f8c-4bee-9453-93312f12f0d3\" (UID: \"3ea20541-9f8c-4bee-9453-93312f12f0d3\") "
Sep 11 00:00:27.855125 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:27.855088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea20541-9f8c-4bee-9453-93312f12f0d3-kube-api-access-62v8k" (OuterVolumeSpecName: "kube-api-access-62v8k") pod "3ea20541-9f8c-4bee-9453-93312f12f0d3" (UID: "3ea20541-9f8c-4bee-9453-93312f12f0d3"). InnerVolumeSpecName "kube-api-access-62v8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:27.954558 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:27.954531    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62v8k\" (UniqueName: \"kubernetes.io/projected/3ea20541-9f8c-4bee-9453-93312f12f0d3-kube-api-access-62v8k\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:28.156644 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:28.156575    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:28.558435 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:28.558408    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ea20541-9f8c-4bee-9453-93312f12f0d3" containerID="cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5" exitCode=137
Sep 11 00:00:28.558863 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:28.558448    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:28.558863 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:28.558508    2562 scope.go:117] "RemoveContainer" containerID="cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5"
Sep 11 00:00:28.570916 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:28.570886    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-d490a5\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="3ea20541-9f8c-4bee-9453-93312f12f0d3" pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:28.571436 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:28.571417    2562 scope.go:117] "RemoveContainer" containerID="cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5"
Sep 11 00:00:28.571703 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:28.571683    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5\": container with ID starting with cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5 not found: ID does not exist" containerID="cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5"
Sep 11 00:00:28.571794 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:28.571709    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5"} err="failed to get container status \"cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5\": rpc error: code = NotFound desc = could not find container \"cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5\": container with ID starting with cd1c18f8ce48e18d2874a02b99a040d3d2097aa812e1510285df0a3c32e9dda5 not found: ID does not exist"
Sep 11 00:00:28.656042 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:28.656018    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea20541-9f8c-4bee-9453-93312f12f0d3" path="/var/lib/kubelet/pods/3ea20541-9f8c-4bee-9453-93312f12f0d3/volumes"
Sep 11 00:00:28.657209 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:28.657187    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-d490a5\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="3ea20541-9f8c-4bee-9453-93312f12f0d3" pod="opendatahub/test-curl-7692-d490a5"
Sep 11 00:00:29.027584 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:29.027553    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:29.723259 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:29.723238    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-61e3f2"
Sep 11 00:00:29.769155 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:29.769127    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25j24\" (UniqueName: \"kubernetes.io/projected/61b02256-66a9-4854-ab3e-72df6ebb7ef7-kube-api-access-25j24\") pod \"61b02256-66a9-4854-ab3e-72df6ebb7ef7\" (UID: \"61b02256-66a9-4854-ab3e-72df6ebb7ef7\") "
Sep 11 00:00:29.771095 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:29.771071    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b02256-66a9-4854-ab3e-72df6ebb7ef7-kube-api-access-25j24" (OuterVolumeSpecName: "kube-api-access-25j24") pod "61b02256-66a9-4854-ab3e-72df6ebb7ef7" (UID: "61b02256-66a9-4854-ab3e-72df6ebb7ef7"). InnerVolumeSpecName "kube-api-access-25j24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:29.870116 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:29.870063    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25j24\" (UniqueName: \"kubernetes.io/projected/61b02256-66a9-4854-ab3e-72df6ebb7ef7-kube-api-access-25j24\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:30.570559 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:30.570526    2562 generic.go:356] "Generic (PLEG): container finished" podID="61b02256-66a9-4854-ab3e-72df6ebb7ef7" containerID="cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3" exitCode=137
Sep 11 00:00:30.570738 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:30.570566    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-61e3f2"
Sep 11 00:00:30.570738 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:30.570615    2562 scope.go:117] "RemoveContainer" containerID="cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3"
Sep 11 00:00:30.579256 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:30.579239    2562 scope.go:117] "RemoveContainer" containerID="cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3"
Sep 11 00:00:30.579514 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:30.579499    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3\": container with ID starting with cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3 not found: ID does not exist" containerID="cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3"
Sep 11 00:00:30.579565 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:30.579522    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3"} err="failed to get container status \"cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3\": rpc error: code = NotFound desc = could not find container \"cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3\": container with ID starting with cbd4727db12deae8a1e0c12c4b5c027a02476b72d06cdf09753c6178a0bf46c3 not found: ID does not exist"
Sep 11 00:00:30.581162 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:30.581142    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-61e3f2\" is forbidden: User \"system:node:ip-10-0-137-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-218.ec2.internal' and this object" podUID="61b02256-66a9-4854-ab3e-72df6ebb7ef7" pod="opendatahub/test-curl-7692-61e3f2"
Sep 11 00:00:30.655554 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:30.655529    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b02256-66a9-4854-ab3e-72df6ebb7ef7" path="/var/lib/kubelet/pods/61b02256-66a9-4854-ab3e-72df6ebb7ef7/volumes"
Sep 11 00:00:31.135650 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.135615    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"]
Sep 11 00:00:31.136215 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.136198    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b02256-66a9-4854-ab3e-72df6ebb7ef7" containerName="test-curl-7692-61e3f2"
Sep 11 00:00:31.136263 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.136219    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b02256-66a9-4854-ab3e-72df6ebb7ef7" containerName="test-curl-7692-61e3f2"
Sep 11 00:00:31.136349 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.136337    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b02256-66a9-4854-ab3e-72df6ebb7ef7" containerName="test-curl-7692-61e3f2"
Sep 11 00:00:31.140878 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.140861    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.143992 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.143973    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"default-dockercfg-2scc9\""
Sep 11 00:00:31.145941 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.145912    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"kube-root-ca.crt\""
Sep 11 00:00:31.146424 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.146377    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"rate-test-model-d7a539-kserve-self-signed-certs\""
Sep 11 00:00:31.146718 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.146701    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"openshift-service-ca.crt\""
Sep 11 00:00:31.150701 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.150680    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"]
Sep 11 00:00:31.174157 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.174123    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:31.282185 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.282157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jpf7\" (UniqueName: \"kubernetes.io/projected/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-kube-api-access-6jpf7\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.282296 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.282198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-dshm\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.282296 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.282216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tls-certs\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.282296 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.282252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-model-cache\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.282420 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.282299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tmp-dir\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.282420 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.282321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-home\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.383462 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.383434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tmp-dir\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.383462 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.383463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-home\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.383627 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.383498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jpf7\" (UniqueName: \"kubernetes.io/projected/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-kube-api-access-6jpf7\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.383627 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.383535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-dshm\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.383627 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.383553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tls-certs\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.383627 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.383572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-model-cache\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.383977 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.383921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-home\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.384062 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.383976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-model-cache\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.384062 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.384026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tmp-dir\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.385778 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.385729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-dshm\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.385903 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.385879    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tls-certs\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.391670 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.391652    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jpf7\" (UniqueName: \"kubernetes.io/projected/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-kube-api-access-6jpf7\") pod \"rate-test-model-d7a539-kserve-7b779754b7-dqcdk\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.450519 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.450493    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:31.720396 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:31.720371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"]
Sep 11 00:00:31.722324 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:00:31.722297    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod312fd8f6_4e13_4114_8cd1_ae47e7f906d0.slice/crio-96660aecca2d04ceef3c7f4de6cb62d492d28ecb3449a001db2430d22edf9118 WatchSource:0}: Error finding container 96660aecca2d04ceef3c7f4de6cb62d492d28ecb3449a001db2430d22edf9118: Status 404 returned error can't find the container with id 96660aecca2d04ceef3c7f4de6cb62d492d28ecb3449a001db2430d22edf9118
Sep 11 00:00:32.426698 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:32.426664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:32.584500 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:32.584456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk" event={"ID":"312fd8f6-4e13-4114-8cd1-ae47e7f906d0","Type":"ContainerStarted","Data":"5e9c349c608bbbb8a7d13f3b07b5c5ccd5f41d8e2c03d27536da9eab074c431b"}
Sep 11 00:00:32.584667 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:32.584508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk" event={"ID":"312fd8f6-4e13-4114-8cd1-ae47e7f906d0","Type":"ContainerStarted","Data":"96660aecca2d04ceef3c7f4de6cb62d492d28ecb3449a001db2430d22edf9118"}
Sep 11 00:00:32.623626 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:32.623575    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk" podStartSLOduration=1.6235596810000001 podStartE2EDuration="1.623559681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:32.622615247 +0000 UTC m=+1444.453642779" watchObservedRunningTime="2026-09-11 00:00:32.623559681 +0000 UTC m=+1444.454587213"
Sep 11 00:00:36.749272 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:36.749217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:37.288971 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.288939    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"]
Sep 11 00:00:37.289172 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.289147    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh" podUID="53923999-1831-452c-8399-e7e81365abc5" containerName="maas-api" containerID="cri-o://973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d" gracePeriod=30
Sep 11 00:00:37.319526 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.319457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:37.340273 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.340238    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"]
Sep 11 00:00:37.340628 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.340536    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp" podUID="8e1e418b-89a0-430d-b8fa-acece16ee0c4" containerName="payload-processing" containerID="cri-o://6360344f63928105659f8ad5a2a1337a4b301802530bca7b87e0b05b7cc1d4ac" gracePeriod=30
Sep 11 00:00:37.347260 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.347214    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"]
Sep 11 00:00:37.347503 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.347458    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576" podUID="1aaf3e09-3091-4647-bb89-1011cf00d433" containerName="payload-pre-processing" containerID="cri-o://8a6948f5426dad4be105265c7adcbe7ff4d982bf44a015904b0e51c7219adbf4" gracePeriod=30
Sep 11 00:00:37.407603 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.407557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:37.511330 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.511285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:37.591166 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.591120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:37.600907 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.600880    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 11 00:00:37.612000 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.611976    2562 generic.go:356] "Generic (PLEG): container finished" podID="1aaf3e09-3091-4647-bb89-1011cf00d433" containerID="8a6948f5426dad4be105265c7adcbe7ff4d982bf44a015904b0e51c7219adbf4" exitCode=0
Sep 11 00:00:37.612109 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.612046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576" event={"ID":"1aaf3e09-3091-4647-bb89-1011cf00d433","Type":"ContainerDied","Data":"8a6948f5426dad4be105265c7adcbe7ff4d982bf44a015904b0e51c7219adbf4"}
Sep 11 00:00:37.613538 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.613516    2562 generic.go:356] "Generic (PLEG): container finished" podID="53923999-1831-452c-8399-e7e81365abc5" containerID="973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d" exitCode=0
Sep 11 00:00:37.613644 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.613561    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"
Sep 11 00:00:37.613644 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.613601    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh" event={"ID":"53923999-1831-452c-8399-e7e81365abc5","Type":"ContainerDied","Data":"973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d"}
Sep 11 00:00:37.613752 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.613654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh" event={"ID":"53923999-1831-452c-8399-e7e81365abc5","Type":"ContainerDied","Data":"b0c48219ded1799edc2b9d81117f56404a60bb0495dabc6c9fb33eb759c194fb"}
Sep 11 00:00:37.613752 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.613682    2562 scope.go:117] "RemoveContainer" containerID="973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d"
Sep 11 00:00:37.615348 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.615324    2562 generic.go:356] "Generic (PLEG): container finished" podID="8e1e418b-89a0-430d-b8fa-acece16ee0c4" containerID="6360344f63928105659f8ad5a2a1337a4b301802530bca7b87e0b05b7cc1d4ac" exitCode=0
Sep 11 00:00:37.615447 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.615357    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp" event={"ID":"8e1e418b-89a0-430d-b8fa-acece16ee0c4","Type":"ContainerDied","Data":"6360344f63928105659f8ad5a2a1337a4b301802530bca7b87e0b05b7cc1d4ac"}
Sep 11 00:00:37.624194 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.624158    2562 scope.go:117] "RemoveContainer" containerID="973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d"
Sep 11 00:00:37.624497 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:37.624475    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d\": container with ID starting with 973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d not found: ID does not exist" containerID="973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d"
Sep 11 00:00:37.624600 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.624503    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d"} err="failed to get container status \"973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d\": rpc error: code = NotFound desc = could not find container \"973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d\": container with ID starting with 973590e9f3990e5855e4cb0de67530090a36ddc04477cd73e75c4b81f313703d not found: ID does not exist"
Sep 11 00:00:37.642652 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.642623    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 11 00:00:37.645066 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.645047    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 11 00:00:37.741979 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.741951    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaf3e09-3091-4647-bb89-1011cf00d433-plugins-config-volume\") pod \"1aaf3e09-3091-4647-bb89-1011cf00d433\" (UID: \"1aaf3e09-3091-4647-bb89-1011cf00d433\") "
Sep 11 00:00:37.742153 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742002    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e418b-89a0-430d-b8fa-acece16ee0c4-plugins-config-volume\") pod \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\" (UID: \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\") "
Sep 11 00:00:37.742153 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742026    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwgwm\" (UniqueName: \"kubernetes.io/projected/53923999-1831-452c-8399-e7e81365abc5-kube-api-access-dwgwm\") pod \"53923999-1831-452c-8399-e7e81365abc5\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") "
Sep 11 00:00:37.742153 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742044    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls\") pod \"53923999-1831-452c-8399-e7e81365abc5\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") "
Sep 11 00:00:37.742153 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742068    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-metrics-certs\") pod \"53923999-1831-452c-8399-e7e81365abc5\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") "
Sep 11 00:00:37.742153 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742086    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j77d\" (UniqueName: \"kubernetes.io/projected/1aaf3e09-3091-4647-bb89-1011cf00d433-kube-api-access-2j77d\") pod \"1aaf3e09-3091-4647-bb89-1011cf00d433\" (UID: \"1aaf3e09-3091-4647-bb89-1011cf00d433\") "
Sep 11 00:00:37.742153 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742132    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53923999-1831-452c-8399-e7e81365abc5-openshift-service-ca\") pod \"53923999-1831-452c-8399-e7e81365abc5\" (UID: \"53923999-1831-452c-8399-e7e81365abc5\") "
Sep 11 00:00:37.742469 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742214    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb7x2\" (UniqueName: \"kubernetes.io/projected/8e1e418b-89a0-430d-b8fa-acece16ee0c4-kube-api-access-bb7x2\") pod \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\" (UID: \"8e1e418b-89a0-430d-b8fa-acece16ee0c4\") "
Sep 11 00:00:37.742469 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742371    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aaf3e09-3091-4647-bb89-1011cf00d433-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1aaf3e09-3091-4647-bb89-1011cf00d433" (UID: "1aaf3e09-3091-4647-bb89-1011cf00d433"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:37.742612 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742585    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1e418b-89a0-430d-b8fa-acece16ee0c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e1e418b-89a0-430d-b8fa-acece16ee0c4" (UID: "8e1e418b-89a0-430d-b8fa-acece16ee0c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:37.742686 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742674    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaf3e09-3091-4647-bb89-1011cf00d433-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:37.742740 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742694    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e418b-89a0-430d-b8fa-acece16ee0c4-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:37.742837 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.742810    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53923999-1831-452c-8399-e7e81365abc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53923999-1831-452c-8399-e7e81365abc5" (UID: "53923999-1831-452c-8399-e7e81365abc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:37.744414 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.744391    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaf3e09-3091-4647-bb89-1011cf00d433-kube-api-access-2j77d" (OuterVolumeSpecName: "kube-api-access-2j77d") pod "1aaf3e09-3091-4647-bb89-1011cf00d433" (UID: "1aaf3e09-3091-4647-bb89-1011cf00d433"). InnerVolumeSpecName "kube-api-access-2j77d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:37.744677 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.744659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53923999-1831-452c-8399-e7e81365abc5-kube-api-access-dwgwm" (OuterVolumeSpecName: "kube-api-access-dwgwm") pod "53923999-1831-452c-8399-e7e81365abc5" (UID: "53923999-1831-452c-8399-e7e81365abc5"). InnerVolumeSpecName "kube-api-access-dwgwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:37.744748 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.744658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53923999-1831-452c-8399-e7e81365abc5" (UID: "53923999-1831-452c-8399-e7e81365abc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:00:37.744748 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.744674    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1e418b-89a0-430d-b8fa-acece16ee0c4-kube-api-access-bb7x2" (OuterVolumeSpecName: "kube-api-access-bb7x2") pod "8e1e418b-89a0-430d-b8fa-acece16ee0c4" (UID: "8e1e418b-89a0-430d-b8fa-acece16ee0c4"). InnerVolumeSpecName "kube-api-access-bb7x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:37.744820 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.744795    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "53923999-1831-452c-8399-e7e81365abc5" (UID: "53923999-1831-452c-8399-e7e81365abc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:00:37.843394 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.843333    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:37.843394 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.843356    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j77d\" (UniqueName: \"kubernetes.io/projected/1aaf3e09-3091-4647-bb89-1011cf00d433-kube-api-access-2j77d\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:37.843394 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.843366    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53923999-1831-452c-8399-e7e81365abc5-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:37.843394 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.843374    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb7x2\" (UniqueName: \"kubernetes.io/projected/8e1e418b-89a0-430d-b8fa-acece16ee0c4-kube-api-access-bb7x2\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:37.843394 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.843382    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwgwm\" (UniqueName: \"kubernetes.io/projected/53923999-1831-452c-8399-e7e81365abc5-kube-api-access-dwgwm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:37.843814 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.843413    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53923999-1831-452c-8399-e7e81365abc5-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:37.856054 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.856030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:37.935771 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.935747    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"]
Sep 11 00:00:37.939080 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:37.939055    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b19d25-6ddb8fbf57-242xh"]
Sep 11 00:00:38.623375 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.623289    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"
Sep 11 00:00:38.623375 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.623297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp" event={"ID":"8e1e418b-89a0-430d-b8fa-acece16ee0c4","Type":"ContainerDied","Data":"1e3d84ce7d26a592f711099ef284944f23c8f010cb0a9a48e503bd871e185107"}
Sep 11 00:00:38.623375 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.623360    2562 scope.go:117] "RemoveContainer" containerID="6360344f63928105659f8ad5a2a1337a4b301802530bca7b87e0b05b7cc1d4ac"
Sep 11 00:00:38.624648 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.624632    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"
Sep 11 00:00:38.624719 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.624643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576" event={"ID":"1aaf3e09-3091-4647-bb89-1011cf00d433","Type":"ContainerDied","Data":"7c5f40e0b9e26e253a9e4263bb2d77a058c35397ed6a15a29e4add4f9ca044bc"}
Sep 11 00:00:38.633631 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.633611    2562 scope.go:117] "RemoveContainer" containerID="8a6948f5426dad4be105265c7adcbe7ff4d982bf44a015904b0e51c7219adbf4"
Sep 11 00:00:38.645256 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.645233    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"]
Sep 11 00:00:38.655908 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.655889    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53923999-1831-452c-8399-e7e81365abc5" path="/var/lib/kubelet/pods/53923999-1831-452c-8399-e7e81365abc5/volumes"
Sep 11 00:00:38.656184 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.656170    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b19d25-76dfd7c9c9-6ckdp"]
Sep 11 00:00:38.660851 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.660831    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"]
Sep 11 00:00:38.663349 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:38.663327    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b19d25-6747fc684b-5w576"]
Sep 11 00:00:40.661586 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:40.661541    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aaf3e09-3091-4647-bb89-1011cf00d433" path="/var/lib/kubelet/pods/1aaf3e09-3091-4647-bb89-1011cf00d433/volumes"
Sep 11 00:00:40.662069 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:40.661902    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1e418b-89a0-430d-b8fa-acece16ee0c4" path="/var/lib/kubelet/pods/8e1e418b-89a0-430d-b8fa-acece16ee0c4/volumes"
Sep 11 00:00:40.681863 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:40.681837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:41.451048 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:41.451002    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:41.451048 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:41.451056    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:41.463676 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:41.463654    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:41.652284 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:41.652257    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:00:43.822655 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:43.822622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:43.923092 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:43.923063    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:44.538937 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:44.538890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:48.487087 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:48.487052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:54.832366 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:54.832329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:54.868186 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:54.868147    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"]
Sep 11 00:00:54.868418 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:54.868380    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r" podUID="2c11fd51-7fc6-448f-816c-f97bfcdc8d1f" containerName="payload-pre-processing" containerID="cri-o://c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272" gracePeriod=30
Sep 11 00:00:54.910013 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:54.909979    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:55.013289 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.013218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:55.083833 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.083764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:55.121722 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.121697    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 11 00:00:55.166938 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.166899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:55.288361 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.288336    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thlxz\" (UniqueName: \"kubernetes.io/projected/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-kube-api-access-thlxz\") pod \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\" (UID: \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\") "
Sep 11 00:00:55.288492 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.288444    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-plugins-config-volume\") pod \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\" (UID: \"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f\") "
Sep 11 00:00:55.288815 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.288792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c11fd51-7fc6-448f-816c-f97bfcdc8d1f" (UID: "2c11fd51-7fc6-448f-816c-f97bfcdc8d1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:55.290306 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.290285    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-kube-api-access-thlxz" (OuterVolumeSpecName: "kube-api-access-thlxz") pod "2c11fd51-7fc6-448f-816c-f97bfcdc8d1f" (UID: "2c11fd51-7fc6-448f-816c-f97bfcdc8d1f"). InnerVolumeSpecName "kube-api-access-thlxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:55.389522 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.389451    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:55.389522 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.389480    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thlxz\" (UniqueName: \"kubernetes.io/projected/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f-kube-api-access-thlxz\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:00:55.459471 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.459439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:55.708628 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.708591    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c11fd51-7fc6-448f-816c-f97bfcdc8d1f" containerID="c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272" exitCode=0
Sep 11 00:00:55.708769 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.708639    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"
Sep 11 00:00:55.708769 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.708673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r" event={"ID":"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f","Type":"ContainerDied","Data":"c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272"}
Sep 11 00:00:55.708769 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.708724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r" event={"ID":"2c11fd51-7fc6-448f-816c-f97bfcdc8d1f","Type":"ContainerDied","Data":"95e9395671fefb03d2d65f06737abab8e76abf9d2d838a5c5fb389b39dafccc0"}
Sep 11 00:00:55.708769 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.708747    2562 scope.go:117] "RemoveContainer" containerID="c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272"
Sep 11 00:00:55.717219 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.717201    2562 scope.go:117] "RemoveContainer" containerID="c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272"
Sep 11 00:00:55.717468 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:00:55.717447    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272\": container with ID starting with c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272 not found: ID does not exist" containerID="c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272"
Sep 11 00:00:55.717513 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.717476    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272"} err="failed to get container status \"c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272\": rpc error: code = NotFound desc = could not find container \"c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272\": container with ID starting with c61f62b7ddf12b2cd71628e48774b36165ebe018ab1b2687af744d629bf57272 not found: ID does not exist"
Sep 11 00:00:55.730723 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.730700    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"]
Sep 11 00:00:55.734241 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:55.734223    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a78403-59556f4d9d-gkc6r"]
Sep 11 00:00:56.656313 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:56.656279    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c11fd51-7fc6-448f-816c-f97bfcdc8d1f" path="/var/lib/kubelet/pods/2c11fd51-7fc6-448f-816c-f97bfcdc8d1f/volumes"
Sep 11 00:00:57.464336 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:57.464283    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:58.941140 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941102    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"]
Sep 11 00:00:58.941701 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941685    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1e418b-89a0-430d-b8fa-acece16ee0c4" containerName="payload-processing"
Sep 11 00:00:58.941774 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941705    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1e418b-89a0-430d-b8fa-acece16ee0c4" containerName="payload-processing"
Sep 11 00:00:58.941774 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941717    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c11fd51-7fc6-448f-816c-f97bfcdc8d1f" containerName="payload-pre-processing"
Sep 11 00:00:58.941774 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941724    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c11fd51-7fc6-448f-816c-f97bfcdc8d1f" containerName="payload-pre-processing"
Sep 11 00:00:58.941774 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941748    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aaf3e09-3091-4647-bb89-1011cf00d433" containerName="payload-pre-processing"
Sep 11 00:00:58.941774 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941758    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aaf3e09-3091-4647-bb89-1011cf00d433" containerName="payload-pre-processing"
Sep 11 00:00:58.941774 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941772    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53923999-1831-452c-8399-e7e81365abc5" containerName="maas-api"
Sep 11 00:00:58.942109 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941781    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53923999-1831-452c-8399-e7e81365abc5" containerName="maas-api"
Sep 11 00:00:58.942109 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941875    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aaf3e09-3091-4647-bb89-1011cf00d433" containerName="payload-pre-processing"
Sep 11 00:00:58.942109 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941886    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1e418b-89a0-430d-b8fa-acece16ee0c4" containerName="payload-processing"
Sep 11 00:00:58.942109 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941897    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c11fd51-7fc6-448f-816c-f97bfcdc8d1f" containerName="payload-pre-processing"
Sep 11 00:00:58.942109 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.941910    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53923999-1831-452c-8399-e7e81365abc5" containerName="maas-api"
Sep 11 00:00:58.945087 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.945066    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:58.948002 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.947979    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"default-dockercfg-jn5c6\""
Sep 11 00:00:58.948119 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.948008    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"openshift-service-ca.crt\""
Sep 11 00:00:58.948997 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.948979    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"rate-test-model-4094ce-kserve-self-signed-certs\""
Sep 11 00:00:58.949087 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.949062    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"kube-root-ca.crt\""
Sep 11 00:00:58.960292 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:58.960269    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"]
Sep 11 00:00:59.001177 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.001148    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:00:59.123173 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.123146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4wz\" (UniqueName: \"kubernetes.io/projected/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-kube-api-access-tn4wz\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.123325 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.123188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-dshm\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.123325 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.123241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tmp-dir\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.123325 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.123279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-model-cache\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.123432 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.123345    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tls-certs\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.123432 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.123376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-home\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.224919 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.224813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-dshm\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.224919 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.224881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tmp-dir\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.225191 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.224955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-model-cache\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.225191 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.225002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tls-certs\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.225191 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.225049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-home\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.225191 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.225160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4wz\" (UniqueName: \"kubernetes.io/projected/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-kube-api-access-tn4wz\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.225385 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.225335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tmp-dir\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.225439 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.225388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-model-cache\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.225494 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.225446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-home\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.227286 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.227258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-dshm\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.227447 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.227432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tls-certs\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.234416 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.234393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4wz\" (UniqueName: \"kubernetes.io/projected/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-kube-api-access-tn4wz\") pod \"rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.255381 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.255360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:00:59.528944 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.528906    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"]
Sep 11 00:00:59.531423 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:00:59.531392    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea9b117a_5fc1_43f2_9d9a_591852fa9f2c.slice/crio-dc33077711f312e0e5f81ee41f575f93c2876cfde9653a21a7db06201af3ea80 WatchSource:0}: Error finding container dc33077711f312e0e5f81ee41f575f93c2876cfde9653a21a7db06201af3ea80: Status 404 returned error can't find the container with id dc33077711f312e0e5f81ee41f575f93c2876cfde9653a21a7db06201af3ea80
Sep 11 00:00:59.734622 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.734583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8" event={"ID":"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c","Type":"ContainerStarted","Data":"faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb"}
Sep 11 00:00:59.734622 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.734625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8" event={"ID":"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c","Type":"ContainerStarted","Data":"dc33077711f312e0e5f81ee41f575f93c2876cfde9653a21a7db06201af3ea80"}
Sep 11 00:00:59.754677 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:00:59.754630    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8" podStartSLOduration=1.754614402 podStartE2EDuration="1.754614402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:59.753140275 +0000 UTC m=+1471.584167830" watchObservedRunningTime="2026-09-11 00:00:59.754614402 +0000 UTC m=+1471.585641969"
Sep 11 00:01:02.100894 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:02.100863    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:02.177531 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:02.177502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:02.367100 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:02.367028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:02.808539 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:02.808505    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:09.256281 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:09.256238    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:01:09.256698 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:09.256323    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:01:09.269178 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:09.269153    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:01:09.793495 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:09.793460    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:01:15.210240 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:15.210206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:28.662174 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:28.662147    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 11 00:01:28.667686 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:28.667663    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 11 00:01:41.796106 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:41.796070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:41.897148 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:41.897109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:51.864330 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:51.864241    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"]
Sep 11 00:01:51.884026 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:51.884001    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:52.816880 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:52.816841    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"]
Sep 11 00:01:52.839500 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:52.839473    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:53.406327 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:53.406285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:53.580713 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:53.580675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:53.818228 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:53.818200    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:56.325268 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:56.325233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:56.419574 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:56.419541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:57.480596 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:57.480565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:57.706063 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:57.706027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:57.806602 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:57.806520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:57.941861 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:57.941829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:58.039709 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.039679    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:01:58.057899 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.057836    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"]
Sep 11 00:01:58.061632 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.061615    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:58.064208 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.064185    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-17664f-dockercfg-8pnk4\""
Sep 11 00:01:58.064208 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.064204    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-17664f\""
Sep 11 00:01:58.073031 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.073009    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"]
Sep 11 00:01:58.083806 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.083784    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"]
Sep 11 00:01:58.087087 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.087069    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:58.098424 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.098397    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"]
Sep 11 00:01:58.216083 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.216044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6vb\" (UniqueName: \"kubernetes.io/projected/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-kube-api-access-ww6vb\") pod \"payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc\" (UID: \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:58.216246 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.216104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc\" (UID: \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:58.216246 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.216156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5jbs\" (UniqueName: \"kubernetes.io/projected/418cd2f8-323f-4ef2-9726-294257dd3374-kube-api-access-b5jbs\") pod \"payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz\" (UID: \"418cd2f8-323f-4ef2-9726-294257dd3374\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:58.216246 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.216198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cd2f8-323f-4ef2-9726-294257dd3374-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz\" (UID: \"418cd2f8-323f-4ef2-9726-294257dd3374\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:58.317389 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.317304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6vb\" (UniqueName: \"kubernetes.io/projected/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-kube-api-access-ww6vb\") pod \"payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc\" (UID: \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:58.317389 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.317358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc\" (UID: \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:58.317389 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.317381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5jbs\" (UniqueName: \"kubernetes.io/projected/418cd2f8-323f-4ef2-9726-294257dd3374-kube-api-access-b5jbs\") pod \"payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz\" (UID: \"418cd2f8-323f-4ef2-9726-294257dd3374\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:58.317631 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.317423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cd2f8-323f-4ef2-9726-294257dd3374-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz\" (UID: \"418cd2f8-323f-4ef2-9726-294257dd3374\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:58.318335 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.318312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc\" (UID: \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:58.318395 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.318312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cd2f8-323f-4ef2-9726-294257dd3374-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz\" (UID: \"418cd2f8-323f-4ef2-9726-294257dd3374\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:58.326299 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.326272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5jbs\" (UniqueName: \"kubernetes.io/projected/418cd2f8-323f-4ef2-9726-294257dd3374-kube-api-access-b5jbs\") pod \"payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz\" (UID: \"418cd2f8-323f-4ef2-9726-294257dd3374\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:58.326410 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.326310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6vb\" (UniqueName: \"kubernetes.io/projected/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-kube-api-access-ww6vb\") pod \"payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc\" (UID: \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:58.372138 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.372115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:58.396809 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.396783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:58.653235 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:01:58.653203    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a837e4_399d_4472_9bd4_6e82b7b5c0a3.slice/crio-e6398e44b147dd8a48b190df0f08b4353ccd69ef398292be9d78183260c320db WatchSource:0}: Error finding container e6398e44b147dd8a48b190df0f08b4353ccd69ef398292be9d78183260c320db: Status 404 returned error can't find the container with id e6398e44b147dd8a48b190df0f08b4353ccd69ef398292be9d78183260c320db
Sep 11 00:01:58.658431 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.658410    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"]
Sep 11 00:01:58.678216 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:58.678194    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"]
Sep 11 00:01:58.684061 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:01:58.684036    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418cd2f8_323f_4ef2_9726_294257dd3374.slice/crio-e269c5ea30c79b4de2d69dff7ff57b44a3754f6edb6aa08c7f3217b244ffb7e1 WatchSource:0}: Error finding container e269c5ea30c79b4de2d69dff7ff57b44a3754f6edb6aa08c7f3217b244ffb7e1: Status 404 returned error can't find the container with id e269c5ea30c79b4de2d69dff7ff57b44a3754f6edb6aa08c7f3217b244ffb7e1
Sep 11 00:01:59.033175 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:59.033117    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc" event={"ID":"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3","Type":"ContainerStarted","Data":"0d01a702b091d544d2bdd5cf42a34fb48958ea12d7bb2d775519a1e58e6fc53a"}
Sep 11 00:01:59.033470 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:59.033442    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:01:59.033577 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:59.033482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc" event={"ID":"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3","Type":"ContainerStarted","Data":"e6398e44b147dd8a48b190df0f08b4353ccd69ef398292be9d78183260c320db"}
Sep 11 00:01:59.034709 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:59.034679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz" event={"ID":"418cd2f8-323f-4ef2-9726-294257dd3374","Type":"ContainerStarted","Data":"b640742340f120f2fd5d688b4c34f364b30b0f41f87d1a470fbfa2cb69d74226"}
Sep 11 00:01:59.034825 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:59.034728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz" event={"ID":"418cd2f8-323f-4ef2-9726-294257dd3374","Type":"ContainerStarted","Data":"e269c5ea30c79b4de2d69dff7ff57b44a3754f6edb6aa08c7f3217b244ffb7e1"}
Sep 11 00:01:59.034825 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:59.034773    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:01:59.049189 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:59.049144    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc" podStartSLOduration=1.049132583 podStartE2EDuration="1.049132583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:01:59.047358308 +0000 UTC m=+1530.878385869" watchObservedRunningTime="2026-09-11 00:01:59.049132583 +0000 UTC m=+1530.880160115"
Sep 11 00:01:59.062192 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:01:59.062157    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz" podStartSLOduration=1.06214597 podStartE2EDuration="1.06214597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:01:59.06164523 +0000 UTC m=+1530.892672787" watchObservedRunningTime="2026-09-11 00:01:59.06214597 +0000 UTC m=+1530.893173544"
Sep 11 00:02:06.883104 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:06.883065    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk" podUID="312fd8f6-4e13-4114-8cd1-ae47e7f906d0" containerName="main" containerID="cri-o://5e9c349c608bbbb8a7d13f3b07b5c5ccd5f41d8e2c03d27536da9eab074c431b" gracePeriod=45
Sep 11 00:02:07.078609 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.078573    2562 generic.go:356] "Generic (PLEG): container finished" podID="312fd8f6-4e13-4114-8cd1-ae47e7f906d0" containerID="5e9c349c608bbbb8a7d13f3b07b5c5ccd5f41d8e2c03d27536da9eab074c431b" exitCode=0
Sep 11 00:02:07.078766 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.078643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk" event={"ID":"312fd8f6-4e13-4114-8cd1-ae47e7f906d0","Type":"ContainerDied","Data":"5e9c349c608bbbb8a7d13f3b07b5c5ccd5f41d8e2c03d27536da9eab074c431b"}
Sep 11 00:02:07.130029 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.130005    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:02:07.189151 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189129    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tmp-dir\") pod \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") "
Sep 11 00:02:07.189294 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189167    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-home\") pod \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") "
Sep 11 00:02:07.189294 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-dshm\") pod \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") "
Sep 11 00:02:07.189294 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189234    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jpf7\" (UniqueName: \"kubernetes.io/projected/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-kube-api-access-6jpf7\") pod \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") "
Sep 11 00:02:07.189294 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189266    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-model-cache\") pod \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") "
Sep 11 00:02:07.189504 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189333    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tls-certs\") pod \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\" (UID: \"312fd8f6-4e13-4114-8cd1-ae47e7f906d0\") "
Sep 11 00:02:07.189504 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189383    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "312fd8f6-4e13-4114-8cd1-ae47e7f906d0" (UID: "312fd8f6-4e13-4114-8cd1-ae47e7f906d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:02:07.189504 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189425    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-home" (OuterVolumeSpecName: "home") pod "312fd8f6-4e13-4114-8cd1-ae47e7f906d0" (UID: "312fd8f6-4e13-4114-8cd1-ae47e7f906d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:02:07.189649 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189570    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:07.189649 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189590    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:07.189649 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.189599    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "312fd8f6-4e13-4114-8cd1-ae47e7f906d0" (UID: "312fd8f6-4e13-4114-8cd1-ae47e7f906d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:02:07.191338 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.191312    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "312fd8f6-4e13-4114-8cd1-ae47e7f906d0" (UID: "312fd8f6-4e13-4114-8cd1-ae47e7f906d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:02:07.191338 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.191312    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-dshm" (OuterVolumeSpecName: "dshm") pod "312fd8f6-4e13-4114-8cd1-ae47e7f906d0" (UID: "312fd8f6-4e13-4114-8cd1-ae47e7f906d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:02:07.191479 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.191313    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-kube-api-access-6jpf7" (OuterVolumeSpecName: "kube-api-access-6jpf7") pod "312fd8f6-4e13-4114-8cd1-ae47e7f906d0" (UID: "312fd8f6-4e13-4114-8cd1-ae47e7f906d0"). InnerVolumeSpecName "kube-api-access-6jpf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:02:07.290093 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.290063    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:07.290093 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.290091    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jpf7\" (UniqueName: \"kubernetes.io/projected/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-kube-api-access-6jpf7\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:07.290280 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.290101    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:07.290280 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.290112    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/312fd8f6-4e13-4114-8cd1-ae47e7f906d0-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:07.836442 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:07.836381    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8" podUID="ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" containerName="main" containerID="cri-o://faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb" gracePeriod=45
Sep 11 00:02:08.082190 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.082165    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:02:08.084115 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.084095    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"
Sep 11 00:02:08.084213 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.084130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk" event={"ID":"312fd8f6-4e13-4114-8cd1-ae47e7f906d0","Type":"ContainerDied","Data":"96660aecca2d04ceef3c7f4de6cb62d492d28ecb3449a001db2430d22edf9118"}
Sep 11 00:02:08.084213 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.084171    2562 scope.go:117] "RemoveContainer" containerID="5e9c349c608bbbb8a7d13f3b07b5c5ccd5f41d8e2c03d27536da9eab074c431b"
Sep 11 00:02:08.085673 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.085647    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" containerID="faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb" exitCode=0
Sep 11 00:02:08.085779 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.085684    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8" event={"ID":"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c","Type":"ContainerDied","Data":"faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb"}
Sep 11 00:02:08.085779 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.085702    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"
Sep 11 00:02:08.085779 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.085714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8" event={"ID":"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c","Type":"ContainerDied","Data":"dc33077711f312e0e5f81ee41f575f93c2876cfde9653a21a7db06201af3ea80"}
Sep 11 00:02:08.093337 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.093322    2562 scope.go:117] "RemoveContainer" containerID="faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb"
Sep 11 00:02:08.097590 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.097561    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-home\") pod \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") "
Sep 11 00:02:08.097681 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.097596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4wz\" (UniqueName: \"kubernetes.io/projected/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-kube-api-access-tn4wz\") pod \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") "
Sep 11 00:02:08.097681 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.097617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tls-certs\") pod \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") "
Sep 11 00:02:08.097817 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.097744    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tmp-dir\") pod \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") "
Sep 11 00:02:08.097817 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.097774    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-model-cache\") pod \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") "
Sep 11 00:02:08.097817 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.097800    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-dshm\") pod \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\" (UID: \"ea9b117a-5fc1-43f2-9d9a-591852fa9f2c\") "
Sep 11 00:02:08.098373 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.097977    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-home" (OuterVolumeSpecName: "home") pod "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" (UID: "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:02:08.098373 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.098095    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:08.098373 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.098086    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" (UID: "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:02:08.098553 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.098405    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" (UID: "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:02:08.100978 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.100393    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" (UID: "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:02:08.100978 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.100918    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-dshm" (OuterVolumeSpecName: "dshm") pod "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" (UID: "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:02:08.101448 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.101424    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-kube-api-access-tn4wz" (OuterVolumeSpecName: "kube-api-access-tn4wz") pod "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" (UID: "ea9b117a-5fc1-43f2-9d9a-591852fa9f2c"). InnerVolumeSpecName "kube-api-access-tn4wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:02:08.103619 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.103597    2562 scope.go:117] "RemoveContainer" containerID="faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb"
Sep 11 00:02:08.104117 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:02:08.104086    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb\": container with ID starting with faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb not found: ID does not exist" containerID="faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb"
Sep 11 00:02:08.104222 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.104126    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb"} err="failed to get container status \"faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb\": rpc error: code = NotFound desc = could not find container \"faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb\": container with ID starting with faae12a07d3d6cb8e6c0cc84d02814e00132774858a7a61621c31eadb89c17cb not found: ID does not exist"
Sep 11 00:02:08.118516 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.118488    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"]
Sep 11 00:02:08.123314 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.123292    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/rate-test-model-d7a539-kserve-7b779754b7-dqcdk"]
Sep 11 00:02:08.198915 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.198889    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:08.198915 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.198914    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:08.199057 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.198922    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:08.199057 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.198956    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4wz\" (UniqueName: \"kubernetes.io/projected/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-kube-api-access-tn4wz\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:08.199057 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.198966    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:02:08.407456 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.407431    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"]
Sep 11 00:02:08.410765 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.410744    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/rate-test-model-4094ce-kserve-84dd86d6f5-6nnd8"]
Sep 11 00:02:08.656917 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.656846    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="312fd8f6-4e13-4114-8cd1-ae47e7f906d0" path="/var/lib/kubelet/pods/312fd8f6-4e13-4114-8cd1-ae47e7f906d0/volumes"
Sep 11 00:02:08.657280 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:08.657267    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" path="/var/lib/kubelet/pods/ea9b117a-5fc1-43f2-9d9a-591852fa9f2c/volumes"
Sep 11 00:02:10.043488 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:10.043463    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:02:11.048483 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:11.048455    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:02:21.318047 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:21.318010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:23.713017 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:23.712980    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:25.536251 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:25.536214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:25.628751 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:25.628718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:26.806435 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.806404    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"]
Sep 11 00:02:26.806804 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.806789    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" containerName="main"
Sep 11 00:02:26.806848 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.806806    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" containerName="main"
Sep 11 00:02:26.806848 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.806814    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312fd8f6-4e13-4114-8cd1-ae47e7f906d0" containerName="main"
Sep 11 00:02:26.806848 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.806819    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="312fd8f6-4e13-4114-8cd1-ae47e7f906d0" containerName="main"
Sep 11 00:02:26.806963 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.806885    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="312fd8f6-4e13-4114-8cd1-ae47e7f906d0" containerName="main"
Sep 11 00:02:26.806963 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.806896    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea9b117a-5fc1-43f2-9d9a-591852fa9f2c" containerName="main"
Sep 11 00:02:26.811146 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.811125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:26.813467 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.813443    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-43f569-serving-cert\""
Sep 11 00:02:26.821208 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.821158    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"]
Sep 11 00:02:26.841622 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.841598    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:26.941552 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.941525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:26.964121 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.964097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-metrics-certs\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:26.964204 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.964126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:26.964204 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.964161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536378c-aa04-467d-97e3-feb97db9d3e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:26.964289 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:26.964233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5g5c\" (UniqueName: \"kubernetes.io/projected/2536378c-aa04-467d-97e3-feb97db9d3e0-kube-api-access-j5g5c\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.051085 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.051058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:27.064839 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.064787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536378c-aa04-467d-97e3-feb97db9d3e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.064839 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.064833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5g5c\" (UniqueName: \"kubernetes.io/projected/2536378c-aa04-467d-97e3-feb97db9d3e0-kube-api-access-j5g5c\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.065028 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.064879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-metrics-certs\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.065028 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.064901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.065487 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.065466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536378c-aa04-467d-97e3-feb97db9d3e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.067235 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.067214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.067312 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.067288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-metrics-certs\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.074536 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.074514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5g5c\" (UniqueName: \"kubernetes.io/projected/2536378c-aa04-467d-97e3-feb97db9d3e0-kube-api-access-j5g5c\") pod \"maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.123347 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.123323    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:27.283743 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.283696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:27.397032 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.396857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:27.399303 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.399279    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"]
Sep 11 00:02:27.401438 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:02:27.401411    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2536378c_aa04_467d_97e3_feb97db9d3e0.slice/crio-1b86224fc77dc772c00646aff5f72386f990a6e4d01dcc5118aaddb81d4b82d7 WatchSource:0}: Error finding container 1b86224fc77dc772c00646aff5f72386f990a6e4d01dcc5118aaddb81d4b82d7: Status 404 returned error can't find the container with id 1b86224fc77dc772c00646aff5f72386f990a6e4d01dcc5118aaddb81d4b82d7
Sep 11 00:02:27.403705 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.403665    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"]
Sep 11 00:02:27.403773 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.403761    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:27.406246 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.406216    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-43f569-dockercfg-hj95d\""
Sep 11 00:02:27.406333 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.406227    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-43f569\""
Sep 11 00:02:27.411052 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.411031    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"]
Sep 11 00:02:27.420372 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.420348    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"]
Sep 11 00:02:27.424469 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.424346    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:27.435614 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.435594    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"]
Sep 11 00:02:27.468058 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.468031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcb2\" (UniqueName: \"kubernetes.io/projected/e95599a9-142f-434c-bec9-473ee207a58e-kube-api-access-lqcb2\") pod \"payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv\" (UID: \"e95599a9-142f-434c-bec9-473ee207a58e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:27.468173 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.468131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95599a9-142f-434c-bec9-473ee207a58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv\" (UID: \"e95599a9-142f-434c-bec9-473ee207a58e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:27.568556 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.568529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73df75ed-2097-49c5-a4f8-b13f3a124ce0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp\" (UID: \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:27.568702 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.568594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95599a9-142f-434c-bec9-473ee207a58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv\" (UID: \"e95599a9-142f-434c-bec9-473ee207a58e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:27.568702 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.568655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9hl\" (UniqueName: \"kubernetes.io/projected/73df75ed-2097-49c5-a4f8-b13f3a124ce0-kube-api-access-2d9hl\") pod \"payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp\" (UID: \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:27.568702 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.568677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcb2\" (UniqueName: \"kubernetes.io/projected/e95599a9-142f-434c-bec9-473ee207a58e-kube-api-access-lqcb2\") pod \"payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv\" (UID: \"e95599a9-142f-434c-bec9-473ee207a58e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:27.569278 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.569256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95599a9-142f-434c-bec9-473ee207a58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv\" (UID: \"e95599a9-142f-434c-bec9-473ee207a58e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:27.576644 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.576624    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcb2\" (UniqueName: \"kubernetes.io/projected/e95599a9-142f-434c-bec9-473ee207a58e-kube-api-access-lqcb2\") pod \"payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv\" (UID: \"e95599a9-142f-434c-bec9-473ee207a58e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:27.670014 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.669954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9hl\" (UniqueName: \"kubernetes.io/projected/73df75ed-2097-49c5-a4f8-b13f3a124ce0-kube-api-access-2d9hl\") pod \"payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp\" (UID: \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:27.670106 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.670028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73df75ed-2097-49c5-a4f8-b13f3a124ce0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp\" (UID: \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:27.670627 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.670604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73df75ed-2097-49c5-a4f8-b13f3a124ce0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp\" (UID: \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:27.678017 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.677994    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9hl\" (UniqueName: \"kubernetes.io/projected/73df75ed-2097-49c5-a4f8-b13f3a124ce0-kube-api-access-2d9hl\") pod \"payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp\" (UID: \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:27.719972 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.719949    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:27.737091 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:27.737068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:28.019443 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:02:28.019412    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95599a9_142f_434c_bec9_473ee207a58e.slice/crio-c7a26ef197ffddfb65a13f04cb09e94de522f0d54a3f060eb2aeb6f2f7b5bb6f WatchSource:0}: Error finding container c7a26ef197ffddfb65a13f04cb09e94de522f0d54a3f060eb2aeb6f2f7b5bb6f: Status 404 returned error can't find the container with id c7a26ef197ffddfb65a13f04cb09e94de522f0d54a3f060eb2aeb6f2f7b5bb6f
Sep 11 00:02:28.019757 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.019663    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"]
Sep 11 00:02:28.037914 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.037891    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"]
Sep 11 00:02:28.039614 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:02:28.039593    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73df75ed_2097_49c5_a4f8_b13f3a124ce0.slice/crio-068473948d43a8c8b275abd69f0a2563bef8aac0fd2b9f1f82dba24af0647f03 WatchSource:0}: Error finding container 068473948d43a8c8b275abd69f0a2563bef8aac0fd2b9f1f82dba24af0647f03: Status 404 returned error can't find the container with id 068473948d43a8c8b275abd69f0a2563bef8aac0fd2b9f1f82dba24af0647f03
Sep 11 00:02:28.194094 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.194057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt" event={"ID":"2536378c-aa04-467d-97e3-feb97db9d3e0","Type":"ContainerStarted","Data":"4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d"}
Sep 11 00:02:28.194273 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.194188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt" event={"ID":"2536378c-aa04-467d-97e3-feb97db9d3e0","Type":"ContainerStarted","Data":"1b86224fc77dc772c00646aff5f72386f990a6e4d01dcc5118aaddb81d4b82d7"}
Sep 11 00:02:28.194273 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.194228    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:28.195978 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.195918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv" event={"ID":"e95599a9-142f-434c-bec9-473ee207a58e","Type":"ContainerStarted","Data":"2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82"}
Sep 11 00:02:28.196117 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.196002    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv" event={"ID":"e95599a9-142f-434c-bec9-473ee207a58e","Type":"ContainerStarted","Data":"c7a26ef197ffddfb65a13f04cb09e94de522f0d54a3f060eb2aeb6f2f7b5bb6f"}
Sep 11 00:02:28.196117 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.196072    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:28.197901 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.197875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp" event={"ID":"73df75ed-2097-49c5-a4f8-b13f3a124ce0","Type":"ContainerStarted","Data":"0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281"}
Sep 11 00:02:28.198012 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.197940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp" event={"ID":"73df75ed-2097-49c5-a4f8-b13f3a124ce0","Type":"ContainerStarted","Data":"068473948d43a8c8b275abd69f0a2563bef8aac0fd2b9f1f82dba24af0647f03"}
Sep 11 00:02:28.198012 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.197966    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:28.215279 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.215222    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt" podStartSLOduration=1.533829817 podStartE2EDuration="2.215203483s" totalImagesPullingTime="681.373666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:02:28.211104586 +0000 UTC m=+1560.042132155" watchObservedRunningTime="2026-09-11 00:02:28.215203483 +0000 UTC m=+1560.046231016"
Sep 11 00:02:28.232493 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.232448    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp" podStartSLOduration=1.232434802 podStartE2EDuration="1.232434802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:02:28.230198405 +0000 UTC m=+1560.061225969" watchObservedRunningTime="2026-09-11 00:02:28.232434802 +0000 UTC m=+1560.063462366"
Sep 11 00:02:28.249139 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:28.249098    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv" podStartSLOduration=1.249084287 podStartE2EDuration="1.249084287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:02:28.247514407 +0000 UTC m=+1560.078541958" watchObservedRunningTime="2026-09-11 00:02:28.249084287 +0000 UTC m=+1560.080111821"
Sep 11 00:02:30.278699 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:30.278664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:34.207071 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:34.207034    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:02:39.203757 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:39.203723    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:02:39.204255 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:39.203985    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:02:43.046575 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:43.046542    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:43.146690 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:43.146659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:48.438517 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:48.438481    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:48.536484 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:48.536451    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:50.520279 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:50.520241    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:02:55.555854 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:55.555822    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5hmzr_eefba04a-fca1-4281-9427-8635b54bd72a/payload-processing/0.log"
Sep 11 00:02:55.672629 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:02:55.672597    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp_73df75ed-2097-49c5-a4f8-b13f3a124ce0/payload-processing/0.log"
Sep 11 00:03:03.628441 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:03.628392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:36.348571 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:36.348499    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz_418cd2f8-323f-4ef2-9726-294257dd3374/payload-processing/0.log"
Sep 11 00:03:36.474054 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:36.474026    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5hmzr_eefba04a-fca1-4281-9427-8635b54bd72a/payload-processing/0.log"
Sep 11 00:03:36.594590 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:36.594570    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp_73df75ed-2097-49c5-a4f8-b13f3a124ce0/payload-processing/0.log"
Sep 11 00:03:36.858806 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:36.858775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:37.165445 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:37.165364    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:37.360239 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:37.360209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:40.190489 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:40.190456    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:40.294765 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:40.294730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:41.446967 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:41.446911    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:41.545211 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:41.545180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:41.605266 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:41.605232    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:41.904162 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:41.904085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:42.002077 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.002039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:03:42.051682 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.051652    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"]
Sep 11 00:03:42.055296 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.055278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:42.059309 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.059267    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ab675d\""
Sep 11 00:03:42.059465 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.059448    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ab675d-dockercfg-kpjbm\""
Sep 11 00:03:42.069183 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.069160    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"]
Sep 11 00:03:42.193870 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.193836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230b78d4-0697-4607-857c-3530c6cfd281-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj\" (UID: \"230b78d4-0697-4607-857c-3530c6cfd281\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:42.194041 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.193872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9bj\" (UniqueName: \"kubernetes.io/projected/230b78d4-0697-4607-857c-3530c6cfd281-kube-api-access-2f9bj\") pod \"payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj\" (UID: \"230b78d4-0697-4607-857c-3530c6cfd281\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:42.295244 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.295216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230b78d4-0697-4607-857c-3530c6cfd281-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj\" (UID: \"230b78d4-0697-4607-857c-3530c6cfd281\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:42.295374 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.295251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9bj\" (UniqueName: \"kubernetes.io/projected/230b78d4-0697-4607-857c-3530c6cfd281-kube-api-access-2f9bj\") pod \"payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj\" (UID: \"230b78d4-0697-4607-857c-3530c6cfd281\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:42.295914 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.295893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230b78d4-0697-4607-857c-3530c6cfd281-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj\" (UID: \"230b78d4-0697-4607-857c-3530c6cfd281\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:42.303247 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.303225    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9bj\" (UniqueName: \"kubernetes.io/projected/230b78d4-0697-4607-857c-3530c6cfd281-kube-api-access-2f9bj\") pod \"payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj\" (UID: \"230b78d4-0697-4607-857c-3530c6cfd281\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:42.365270 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.365248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:42.640590 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:42.640567    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"]
Sep 11 00:03:42.642988 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:03:42.642963    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230b78d4_0697_4607_857c_3530c6cfd281.slice/crio-495991f32536c63d1c823863ad06f168bc3364f018b30714545f6fa8300ac05a WatchSource:0}: Error finding container 495991f32536c63d1c823863ad06f168bc3364f018b30714545f6fa8300ac05a: Status 404 returned error can't find the container with id 495991f32536c63d1c823863ad06f168bc3364f018b30714545f6fa8300ac05a
Sep 11 00:03:43.571089 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:43.571059    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj" event={"ID":"230b78d4-0697-4607-857c-3530c6cfd281","Type":"ContainerStarted","Data":"38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6"}
Sep 11 00:03:43.571290 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:43.571108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj" event={"ID":"230b78d4-0697-4607-857c-3530c6cfd281","Type":"ContainerStarted","Data":"495991f32536c63d1c823863ad06f168bc3364f018b30714545f6fa8300ac05a"}
Sep 11 00:03:43.571290 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:43.571166    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:03:43.589646 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:43.589602    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj" podStartSLOduration=1.589589078 podStartE2EDuration="1.589589078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:43.588245289 +0000 UTC m=+1635.419272842" watchObservedRunningTime="2026-09-11 00:03:43.589589078 +0000 UTC m=+1635.420616623"
Sep 11 00:03:54.577844 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:03:54.577810    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:04:05.117175 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:05.117136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:11.936726 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:11.936693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:11.961268 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:11.961240    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"]
Sep 11 00:04:11.961464 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:11.961442    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj" podUID="230b78d4-0697-4607-857c-3530c6cfd281" containerName="payload-processing" containerID="cri-o://38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6" gracePeriod=30
Sep 11 00:04:12.035810 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.035774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:12.121295 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.121265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:12.187604 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.187529    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:12.210315 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.210296    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:04:12.247644 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.247618    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f9bj\" (UniqueName: \"kubernetes.io/projected/230b78d4-0697-4607-857c-3530c6cfd281-kube-api-access-2f9bj\") pod \"230b78d4-0697-4607-857c-3530c6cfd281\" (UID: \"230b78d4-0697-4607-857c-3530c6cfd281\") "
Sep 11 00:04:12.247763 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.247664    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230b78d4-0697-4607-857c-3530c6cfd281-plugins-config-volume\") pod \"230b78d4-0697-4607-857c-3530c6cfd281\" (UID: \"230b78d4-0697-4607-857c-3530c6cfd281\") "
Sep 11 00:04:12.248111 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.248088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230b78d4-0697-4607-857c-3530c6cfd281-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "230b78d4-0697-4607-857c-3530c6cfd281" (UID: "230b78d4-0697-4607-857c-3530c6cfd281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:04:12.249590 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.249568    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230b78d4-0697-4607-857c-3530c6cfd281-kube-api-access-2f9bj" (OuterVolumeSpecName: "kube-api-access-2f9bj") pod "230b78d4-0697-4607-857c-3530c6cfd281" (UID: "230b78d4-0697-4607-857c-3530c6cfd281"). InnerVolumeSpecName "kube-api-access-2f9bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:04:12.349165 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.349141    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9bj\" (UniqueName: \"kubernetes.io/projected/230b78d4-0697-4607-857c-3530c6cfd281-kube-api-access-2f9bj\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:04:12.349165 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.349163    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230b78d4-0697-4607-857c-3530c6cfd281-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:04:12.491980 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.491840    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:12.713995 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.713967    2562 generic.go:356] "Generic (PLEG): container finished" podID="230b78d4-0697-4607-857c-3530c6cfd281" containerID="38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6" exitCode=0
Sep 11 00:04:12.714149 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.714008    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"
Sep 11 00:04:12.714149 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.714057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj" event={"ID":"230b78d4-0697-4607-857c-3530c6cfd281","Type":"ContainerDied","Data":"38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6"}
Sep 11 00:04:12.717556 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.714500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj" event={"ID":"230b78d4-0697-4607-857c-3530c6cfd281","Type":"ContainerDied","Data":"495991f32536c63d1c823863ad06f168bc3364f018b30714545f6fa8300ac05a"}
Sep 11 00:04:12.717556 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.714568    2562 scope.go:117] "RemoveContainer" containerID="38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6"
Sep 11 00:04:12.729298 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.729281    2562 scope.go:117] "RemoveContainer" containerID="38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6"
Sep 11 00:04:12.729577 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:04:12.729559    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6\": container with ID starting with 38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6 not found: ID does not exist" containerID="38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6"
Sep 11 00:04:12.729622 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.729585    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6"} err="failed to get container status \"38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6\": rpc error: code = NotFound desc = could not find container \"38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6\": container with ID starting with 38a42fcce2cf4cd384895b5188faead8011d79880c4e71daafac45574c689ea6 not found: ID does not exist"
Sep 11 00:04:12.734420 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.734398    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"]
Sep 11 00:04:12.737675 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:12.737653    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ab675d-6c6bbbbb85-dw4dj"]
Sep 11 00:04:14.657429 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:14.657392    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230b78d4-0697-4607-857c-3530c6cfd281" path="/var/lib/kubelet/pods/230b78d4-0697-4607-857c-3530c6cfd281/volumes"
Sep 11 00:04:18.342001 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:18.341956    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:18.403831 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:18.403791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:18.534209 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:18.534176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:22.232148 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:22.232119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:37.586876 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.586839    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:37.619186 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.619156    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"]
Sep 11 00:04:37.619396 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.619373    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz" podUID="418cd2f8-323f-4ef2-9726-294257dd3374" containerName="payload-processing" containerID="cri-o://b640742340f120f2fd5d688b4c34f364b30b0f41f87d1a470fbfa2cb69d74226" gracePeriod=30
Sep 11 00:04:37.628812 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.628785    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"]
Sep 11 00:04:37.629126 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.629017    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc" podUID="a4a837e4-399d-4472-9bd4-6e82b7b5c0a3" containerName="payload-pre-processing" containerID="cri-o://0d01a702b091d544d2bdd5cf42a34fb48958ea12d7bb2d775519a1e58e6fc53a" gracePeriod=30
Sep 11 00:04:37.664658 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.664625    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:37.794962 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.794915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:37.837550 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.837482    2562 generic.go:356] "Generic (PLEG): container finished" podID="418cd2f8-323f-4ef2-9726-294257dd3374" containerID="b640742340f120f2fd5d688b4c34f364b30b0f41f87d1a470fbfa2cb69d74226" exitCode=0
Sep 11 00:04:37.837674 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.837557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz" event={"ID":"418cd2f8-323f-4ef2-9726-294257dd3374","Type":"ContainerDied","Data":"b640742340f120f2fd5d688b4c34f364b30b0f41f87d1a470fbfa2cb69d74226"}
Sep 11 00:04:37.839502 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.839471    2562 generic.go:356] "Generic (PLEG): container finished" podID="a4a837e4-399d-4472-9bd4-6e82b7b5c0a3" containerID="0d01a702b091d544d2bdd5cf42a34fb48958ea12d7bb2d775519a1e58e6fc53a" exitCode=0
Sep 11 00:04:37.839617 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.839558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc" event={"ID":"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3","Type":"ContainerDied","Data":"0d01a702b091d544d2bdd5cf42a34fb48958ea12d7bb2d775519a1e58e6fc53a"}
Sep 11 00:04:37.897239 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.897215    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:04:37.918842 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:37.918823    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:04:38.010643 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.010608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:38.062949 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.062893    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5jbs\" (UniqueName: \"kubernetes.io/projected/418cd2f8-323f-4ef2-9726-294257dd3374-kube-api-access-b5jbs\") pod \"418cd2f8-323f-4ef2-9726-294257dd3374\" (UID: \"418cd2f8-323f-4ef2-9726-294257dd3374\") "
Sep 11 00:04:38.063112 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.062957    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-plugins-config-volume\") pod \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\" (UID: \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\") "
Sep 11 00:04:38.063112 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.063045    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww6vb\" (UniqueName: \"kubernetes.io/projected/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-kube-api-access-ww6vb\") pod \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\" (UID: \"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3\") "
Sep 11 00:04:38.063255 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.063175    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cd2f8-323f-4ef2-9726-294257dd3374-plugins-config-volume\") pod \"418cd2f8-323f-4ef2-9726-294257dd3374\" (UID: \"418cd2f8-323f-4ef2-9726-294257dd3374\") "
Sep 11 00:04:38.063487 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.063460    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4a837e4-399d-4472-9bd4-6e82b7b5c0a3" (UID: "a4a837e4-399d-4472-9bd4-6e82b7b5c0a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:04:38.063591 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.063539    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418cd2f8-323f-4ef2-9726-294257dd3374-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "418cd2f8-323f-4ef2-9726-294257dd3374" (UID: "418cd2f8-323f-4ef2-9726-294257dd3374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:04:38.065122 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.065103    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418cd2f8-323f-4ef2-9726-294257dd3374-kube-api-access-b5jbs" (OuterVolumeSpecName: "kube-api-access-b5jbs") pod "418cd2f8-323f-4ef2-9726-294257dd3374" (UID: "418cd2f8-323f-4ef2-9726-294257dd3374"). InnerVolumeSpecName "kube-api-access-b5jbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:04:38.065206 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.065131    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-kube-api-access-ww6vb" (OuterVolumeSpecName: "kube-api-access-ww6vb") pod "a4a837e4-399d-4472-9bd4-6e82b7b5c0a3" (UID: "a4a837e4-399d-4472-9bd4-6e82b7b5c0a3"). InnerVolumeSpecName "kube-api-access-ww6vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:04:38.164809 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.164725    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww6vb\" (UniqueName: \"kubernetes.io/projected/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-kube-api-access-ww6vb\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:04:38.164809 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.164756    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cd2f8-323f-4ef2-9726-294257dd3374-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:04:38.164809 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.164769    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5jbs\" (UniqueName: \"kubernetes.io/projected/418cd2f8-323f-4ef2-9726-294257dd3374-kube-api-access-b5jbs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:04:38.164809 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.164781    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:04:38.190604 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.190574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:38.845790 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.845767    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"
Sep 11 00:04:38.846243 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.845784    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz" event={"ID":"418cd2f8-323f-4ef2-9726-294257dd3374","Type":"ContainerDied","Data":"e269c5ea30c79b4de2d69dff7ff57b44a3754f6edb6aa08c7f3217b244ffb7e1"}
Sep 11 00:04:38.846243 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.845841    2562 scope.go:117] "RemoveContainer" containerID="b640742340f120f2fd5d688b4c34f364b30b0f41f87d1a470fbfa2cb69d74226"
Sep 11 00:04:38.847367 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.847340    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc" event={"ID":"a4a837e4-399d-4472-9bd4-6e82b7b5c0a3","Type":"ContainerDied","Data":"e6398e44b147dd8a48b190df0f08b4353ccd69ef398292be9d78183260c320db"}
Sep 11 00:04:38.847367 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.847367    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"
Sep 11 00:04:38.854710 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.854695    2562 scope.go:117] "RemoveContainer" containerID="0d01a702b091d544d2bdd5cf42a34fb48958ea12d7bb2d775519a1e58e6fc53a"
Sep 11 00:04:38.864606 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.864583    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"]
Sep 11 00:04:38.868622 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.868590    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17664f-7b8cc588bb-fn6nc"]
Sep 11 00:04:38.878713 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.878689    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"]
Sep 11 00:04:38.885255 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:38.885203    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17664f-85599b85fc-4xvdz"]
Sep 11 00:04:40.655725 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:40.655692    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="418cd2f8-323f-4ef2-9726-294257dd3374" path="/var/lib/kubelet/pods/418cd2f8-323f-4ef2-9726-294257dd3374/volumes"
Sep 11 00:04:40.656120 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:40.656069    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a837e4-399d-4472-9bd4-6e82b7b5c0a3" path="/var/lib/kubelet/pods/a4a837e4-399d-4472-9bd4-6e82b7b5c0a3/volumes"
Sep 11 00:04:42.131898 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:42.131866    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:04:43.735397 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:04:43.735349    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:19.408630 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:19.408595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:20.367865 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:20.367834    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:25.799911 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:25.799881    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:25.858243 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:25.858217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:26.073196 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:26.073113    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:26.703297 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:26.703260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:33.319765 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:33.319726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:36.553380 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:36.553345    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:37.086343 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.086287    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"]
Sep 11 00:05:37.086549 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.086521    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt" podUID="2536378c-aa04-467d-97e3-feb97db9d3e0" containerName="maas-api" containerID="cri-o://4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d" gracePeriod=30
Sep 11 00:05:37.109548 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.109522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:37.137905 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.137874    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"]
Sep 11 00:05:37.138154 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.138126    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp" podUID="73df75ed-2097-49c5-a4f8-b13f3a124ce0" containerName="payload-processing" containerID="cri-o://0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281" gracePeriod=30
Sep 11 00:05:37.141857 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.141802    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"]
Sep 11 00:05:37.142071 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.142019    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv" podUID="e95599a9-142f-434c-bec9-473ee207a58e" containerName="payload-pre-processing" containerID="cri-o://2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82" gracePeriod=30
Sep 11 00:05:37.183474 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.183443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:37.253202 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.253173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:37.324774 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.324746    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:37.398715 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.398694    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:05:37.469910 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.469888    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:05:37.472844 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.472826    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:05:37.473885 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.473868    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:37.544806 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.544781    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536378c-aa04-467d-97e3-feb97db9d3e0-openshift-service-ca\") pod \"2536378c-aa04-467d-97e3-feb97db9d3e0\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") "
Sep 11 00:05:37.544920 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.544833    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-metrics-certs\") pod \"2536378c-aa04-467d-97e3-feb97db9d3e0\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") "
Sep 11 00:05:37.544920 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.544890    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5g5c\" (UniqueName: \"kubernetes.io/projected/2536378c-aa04-467d-97e3-feb97db9d3e0-kube-api-access-j5g5c\") pod \"2536378c-aa04-467d-97e3-feb97db9d3e0\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") "
Sep 11 00:05:37.545041 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.545027    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-maas-api-tls\") pod \"2536378c-aa04-467d-97e3-feb97db9d3e0\" (UID: \"2536378c-aa04-467d-97e3-feb97db9d3e0\") "
Sep 11 00:05:37.545178 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.545146    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2536378c-aa04-467d-97e3-feb97db9d3e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2536378c-aa04-467d-97e3-feb97db9d3e0" (UID: "2536378c-aa04-467d-97e3-feb97db9d3e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:05:37.545296 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.545283    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536378c-aa04-467d-97e3-feb97db9d3e0-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:37.547029 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.546999    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2536378c-aa04-467d-97e3-feb97db9d3e0-kube-api-access-j5g5c" (OuterVolumeSpecName: "kube-api-access-j5g5c") pod "2536378c-aa04-467d-97e3-feb97db9d3e0" (UID: "2536378c-aa04-467d-97e3-feb97db9d3e0"). InnerVolumeSpecName "kube-api-access-j5g5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:05:37.547110 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.547022    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2536378c-aa04-467d-97e3-feb97db9d3e0" (UID: "2536378c-aa04-467d-97e3-feb97db9d3e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:05:37.547110 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.547088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2536378c-aa04-467d-97e3-feb97db9d3e0" (UID: "2536378c-aa04-467d-97e3-feb97db9d3e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:05:37.646001 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.645912    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqcb2\" (UniqueName: \"kubernetes.io/projected/e95599a9-142f-434c-bec9-473ee207a58e-kube-api-access-lqcb2\") pod \"e95599a9-142f-434c-bec9-473ee207a58e\" (UID: \"e95599a9-142f-434c-bec9-473ee207a58e\") "
Sep 11 00:05:37.646001 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.645990    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d9hl\" (UniqueName: \"kubernetes.io/projected/73df75ed-2097-49c5-a4f8-b13f3a124ce0-kube-api-access-2d9hl\") pod \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\" (UID: \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\") "
Sep 11 00:05:37.646425 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.646081    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73df75ed-2097-49c5-a4f8-b13f3a124ce0-plugins-config-volume\") pod \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\" (UID: \"73df75ed-2097-49c5-a4f8-b13f3a124ce0\") "
Sep 11 00:05:37.646425 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.646135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95599a9-142f-434c-bec9-473ee207a58e-plugins-config-volume\") pod \"e95599a9-142f-434c-bec9-473ee207a58e\" (UID: \"e95599a9-142f-434c-bec9-473ee207a58e\") "
Sep 11 00:05:37.646425 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.646386    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:37.646425 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.646406    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536378c-aa04-467d-97e3-feb97db9d3e0-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:37.646425 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.646420    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5g5c\" (UniqueName: \"kubernetes.io/projected/2536378c-aa04-467d-97e3-feb97db9d3e0-kube-api-access-j5g5c\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:37.646604 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.646474    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73df75ed-2097-49c5-a4f8-b13f3a124ce0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73df75ed-2097-49c5-a4f8-b13f3a124ce0" (UID: "73df75ed-2097-49c5-a4f8-b13f3a124ce0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:05:37.646604 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.646496    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95599a9-142f-434c-bec9-473ee207a58e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e95599a9-142f-434c-bec9-473ee207a58e" (UID: "e95599a9-142f-434c-bec9-473ee207a58e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:05:37.647676 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.647658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95599a9-142f-434c-bec9-473ee207a58e-kube-api-access-lqcb2" (OuterVolumeSpecName: "kube-api-access-lqcb2") pod "e95599a9-142f-434c-bec9-473ee207a58e" (UID: "e95599a9-142f-434c-bec9-473ee207a58e"). InnerVolumeSpecName "kube-api-access-lqcb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:05:37.647754 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.647735    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73df75ed-2097-49c5-a4f8-b13f3a124ce0-kube-api-access-2d9hl" (OuterVolumeSpecName: "kube-api-access-2d9hl") pod "73df75ed-2097-49c5-a4f8-b13f3a124ce0" (UID: "73df75ed-2097-49c5-a4f8-b13f3a124ce0"). InnerVolumeSpecName "kube-api-access-2d9hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:05:37.704569 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.704541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:37.747820 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.747796    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73df75ed-2097-49c5-a4f8-b13f3a124ce0-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:37.747957 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.747822    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95599a9-142f-434c-bec9-473ee207a58e-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:37.747957 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.747837    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqcb2\" (UniqueName: \"kubernetes.io/projected/e95599a9-142f-434c-bec9-473ee207a58e-kube-api-access-lqcb2\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:37.747957 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:37.747850    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9hl\" (UniqueName: \"kubernetes.io/projected/73df75ed-2097-49c5-a4f8-b13f3a124ce0-kube-api-access-2d9hl\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:38.134045 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.134014    2562 generic.go:356] "Generic (PLEG): container finished" podID="2536378c-aa04-467d-97e3-feb97db9d3e0" containerID="4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d" exitCode=0
Sep 11 00:05:38.134170 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.134060    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"
Sep 11 00:05:38.134170 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.134102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt" event={"ID":"2536378c-aa04-467d-97e3-feb97db9d3e0","Type":"ContainerDied","Data":"4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d"}
Sep 11 00:05:38.134170 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.134158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt" event={"ID":"2536378c-aa04-467d-97e3-feb97db9d3e0","Type":"ContainerDied","Data":"1b86224fc77dc772c00646aff5f72386f990a6e4d01dcc5118aaddb81d4b82d7"}
Sep 11 00:05:38.134329 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.134181    2562 scope.go:117] "RemoveContainer" containerID="4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d"
Sep 11 00:05:38.135586 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.135501    2562 generic.go:356] "Generic (PLEG): container finished" podID="e95599a9-142f-434c-bec9-473ee207a58e" containerID="2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82" exitCode=0
Sep 11 00:05:38.135679 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.135608    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"
Sep 11 00:05:38.135679 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.135654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv" event={"ID":"e95599a9-142f-434c-bec9-473ee207a58e","Type":"ContainerDied","Data":"2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82"}
Sep 11 00:05:38.135793 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.135700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv" event={"ID":"e95599a9-142f-434c-bec9-473ee207a58e","Type":"ContainerDied","Data":"c7a26ef197ffddfb65a13f04cb09e94de522f0d54a3f060eb2aeb6f2f7b5bb6f"}
Sep 11 00:05:38.137535 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.137489    2562 generic.go:356] "Generic (PLEG): container finished" podID="73df75ed-2097-49c5-a4f8-b13f3a124ce0" containerID="0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281" exitCode=0
Sep 11 00:05:38.137535 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.137517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp" event={"ID":"73df75ed-2097-49c5-a4f8-b13f3a124ce0","Type":"ContainerDied","Data":"0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281"}
Sep 11 00:05:38.137727 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.137575    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp" event={"ID":"73df75ed-2097-49c5-a4f8-b13f3a124ce0","Type":"ContainerDied","Data":"068473948d43a8c8b275abd69f0a2563bef8aac0fd2b9f1f82dba24af0647f03"}
Sep 11 00:05:38.137727 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.137526    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"
Sep 11 00:05:38.145007 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.144985    2562 scope.go:117] "RemoveContainer" containerID="4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d"
Sep 11 00:05:38.145293 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:05:38.145271    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d\": container with ID starting with 4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d not found: ID does not exist" containerID="4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d"
Sep 11 00:05:38.145436 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.145298    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d"} err="failed to get container status \"4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d\": rpc error: code = NotFound desc = could not find container \"4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d\": container with ID starting with 4a0a145079d379b46f583e5b67251650eaa8873d2a65817ee1deca8f4c2e911d not found: ID does not exist"
Sep 11 00:05:38.145436 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.145315    2562 scope.go:117] "RemoveContainer" containerID="2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82"
Sep 11 00:05:38.153101 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.153086    2562 scope.go:117] "RemoveContainer" containerID="2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82"
Sep 11 00:05:38.153354 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:05:38.153337    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82\": container with ID starting with 2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82 not found: ID does not exist" containerID="2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82"
Sep 11 00:05:38.153400 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.153360    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82"} err="failed to get container status \"2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82\": rpc error: code = NotFound desc = could not find container \"2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82\": container with ID starting with 2da57f7c91e9f9557e31dc5514581b922f8f5f3d79f599301e6e7b07d8e65c82 not found: ID does not exist"
Sep 11 00:05:38.153400 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.153378    2562 scope.go:117] "RemoveContainer" containerID="0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281"
Sep 11 00:05:38.157793 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.157771    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"]
Sep 11 00:05:38.160597 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.160580    2562 scope.go:117] "RemoveContainer" containerID="0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281"
Sep 11 00:05:38.160840 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:05:38.160824    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281\": container with ID starting with 0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281 not found: ID does not exist" containerID="0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281"
Sep 11 00:05:38.160888 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.160846    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281"} err="failed to get container status \"0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281\": rpc error: code = NotFound desc = could not find container \"0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281\": container with ID starting with 0ed07d51d419223fcca3274638b5fc26cebeb56807c899d5d0e636f2ff6bc281 not found: ID does not exist"
Sep 11 00:05:38.163694 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.163674    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-43f569-6c67667b54-mpbxt"]
Sep 11 00:05:38.173234 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.173215    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"]
Sep 11 00:05:38.176814 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.176794    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f569-556d7948f4-v55zv"]
Sep 11 00:05:38.186623 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.186603    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"]
Sep 11 00:05:38.190205 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.190186    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f569-6984645b89-l8pgp"]
Sep 11 00:05:38.655139 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.655105    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2536378c-aa04-467d-97e3-feb97db9d3e0" path="/var/lib/kubelet/pods/2536378c-aa04-467d-97e3-feb97db9d3e0/volumes"
Sep 11 00:05:38.655519 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.655490    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73df75ed-2097-49c5-a4f8-b13f3a124ce0" path="/var/lib/kubelet/pods/73df75ed-2097-49c5-a4f8-b13f3a124ce0/volumes"
Sep 11 00:05:38.655851 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:38.655835    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e95599a9-142f-434c-bec9-473ee207a58e" path="/var/lib/kubelet/pods/e95599a9-142f-434c-bec9-473ee207a58e/volumes"
Sep 11 00:05:43.462090 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:43.462054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:43.526154 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:43.526126    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:44.009557 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:44.009502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:44.146774 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:44.146718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:54.544263 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.544223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:54.569765 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.569730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"]
Sep 11 00:05:54.570102 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.570038    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f" podUID="00daa5cc-4569-4b43-91f1-45f03df24a49" containerName="payload-pre-processing" containerID="cri-o://b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8" gracePeriod=30
Sep 11 00:05:54.644881 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.644848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:54.721362 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.721329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:54.789288 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.789248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:54.827845 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.827823    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 11 00:05:54.871248 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.871219    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbjhc\" (UniqueName: \"kubernetes.io/projected/00daa5cc-4569-4b43-91f1-45f03df24a49-kube-api-access-nbjhc\") pod \"00daa5cc-4569-4b43-91f1-45f03df24a49\" (UID: \"00daa5cc-4569-4b43-91f1-45f03df24a49\") "
Sep 11 00:05:54.871399 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.871256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00daa5cc-4569-4b43-91f1-45f03df24a49-plugins-config-volume\") pod \"00daa5cc-4569-4b43-91f1-45f03df24a49\" (UID: \"00daa5cc-4569-4b43-91f1-45f03df24a49\") "
Sep 11 00:05:54.871669 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.871645    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00daa5cc-4569-4b43-91f1-45f03df24a49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00daa5cc-4569-4b43-91f1-45f03df24a49" (UID: "00daa5cc-4569-4b43-91f1-45f03df24a49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:05:54.873222 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.873201    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00daa5cc-4569-4b43-91f1-45f03df24a49-kube-api-access-nbjhc" (OuterVolumeSpecName: "kube-api-access-nbjhc") pod "00daa5cc-4569-4b43-91f1-45f03df24a49" (UID: "00daa5cc-4569-4b43-91f1-45f03df24a49"). InnerVolumeSpecName "kube-api-access-nbjhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:05:54.972433 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.972408    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbjhc\" (UniqueName: \"kubernetes.io/projected/00daa5cc-4569-4b43-91f1-45f03df24a49-kube-api-access-nbjhc\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:54.972433 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:54.972432    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00daa5cc-4569-4b43-91f1-45f03df24a49-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:05:55.092111 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.092020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:05:55.222144 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.222115    2562 generic.go:356] "Generic (PLEG): container finished" podID="00daa5cc-4569-4b43-91f1-45f03df24a49" containerID="b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8" exitCode=0
Sep 11 00:05:55.222258 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.222163    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"
Sep 11 00:05:55.222258 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.222165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f" event={"ID":"00daa5cc-4569-4b43-91f1-45f03df24a49","Type":"ContainerDied","Data":"b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8"}
Sep 11 00:05:55.222258 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.222220    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f" event={"ID":"00daa5cc-4569-4b43-91f1-45f03df24a49","Type":"ContainerDied","Data":"86d53a9fa37fca171e4cb5631976d3550e68ced18f29d8b03fd92e7b76907c8e"}
Sep 11 00:05:55.222258 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.222242    2562 scope.go:117] "RemoveContainer" containerID="b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8"
Sep 11 00:05:55.231257 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.231241    2562 scope.go:117] "RemoveContainer" containerID="b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8"
Sep 11 00:05:55.231552 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:05:55.231532    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8\": container with ID starting with b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8 not found: ID does not exist" containerID="b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8"
Sep 11 00:05:55.231617 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.231560    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8"} err="failed to get container status \"b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8\": rpc error: code = NotFound desc = could not find container \"b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8\": container with ID starting with b30ea2926c1ed89b52cf022d805bab57dc6ed83a09e32af31101e229fe1f09c8 not found: ID does not exist"
Sep 11 00:05:55.242604 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.242580    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"]
Sep 11 00:05:55.246781 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:55.246763    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d7a539-78c78c947d-rbc8f"]
Sep 11 00:05:56.654830 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:05:56.654793    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00daa5cc-4569-4b43-91f1-45f03df24a49" path="/var/lib/kubelet/pods/00daa5cc-4569-4b43-91f1-45f03df24a49/volumes"
Sep 11 00:06:01.177110 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:01.177075    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:01.233599 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:01.233572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:01.412182 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:01.412146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:02.025918 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:02.025884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:12.226785 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.226753    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"]
Sep 11 00:06:12.227296 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.227088    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw" podUID="10093fb8-71d1-49b6-aaba-d538110c1ca3" containerName="maas-api" containerID="cri-o://993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65" gracePeriod=30
Sep 11 00:06:12.266492 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.266465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:12.294480 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.294450    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"]
Sep 11 00:06:12.295159 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.295117    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl" podUID="6963e146-5757-40e2-9a78-64e76533402c" containerName="payload-pre-processing" containerID="cri-o://f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095" gracePeriod=30
Sep 11 00:06:12.345354 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.345320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:12.446331 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.446244    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:12.542826 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.542797    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 11 00:06:12.545401 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.545377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:12.574347 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.574324    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 11 00:06:12.611639 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.611617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66kzs\" (UniqueName: \"kubernetes.io/projected/10093fb8-71d1-49b6-aaba-d538110c1ca3-kube-api-access-66kzs\") pod \"10093fb8-71d1-49b6-aaba-d538110c1ca3\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") "
Sep 11 00:06:12.611770 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.611655    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtl9b\" (UniqueName: \"kubernetes.io/projected/6963e146-5757-40e2-9a78-64e76533402c-kube-api-access-gtl9b\") pod \"6963e146-5757-40e2-9a78-64e76533402c\" (UID: \"6963e146-5757-40e2-9a78-64e76533402c\") "
Sep 11 00:06:12.611770 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.611685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls\") pod \"10093fb8-71d1-49b6-aaba-d538110c1ca3\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") "
Sep 11 00:06:12.611770 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.611734    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-metrics-certs\") pod \"10093fb8-71d1-49b6-aaba-d538110c1ca3\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") "
Sep 11 00:06:12.611770 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.611756    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10093fb8-71d1-49b6-aaba-d538110c1ca3-openshift-service-ca\") pod \"10093fb8-71d1-49b6-aaba-d538110c1ca3\" (UID: \"10093fb8-71d1-49b6-aaba-d538110c1ca3\") "
Sep 11 00:06:12.612024 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.611822    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6963e146-5757-40e2-9a78-64e76533402c-plugins-config-volume\") pod \"6963e146-5757-40e2-9a78-64e76533402c\" (UID: \"6963e146-5757-40e2-9a78-64e76533402c\") "
Sep 11 00:06:12.612251 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.612207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10093fb8-71d1-49b6-aaba-d538110c1ca3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10093fb8-71d1-49b6-aaba-d538110c1ca3" (UID: "10093fb8-71d1-49b6-aaba-d538110c1ca3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:06:12.612447 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.612429    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6963e146-5757-40e2-9a78-64e76533402c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6963e146-5757-40e2-9a78-64e76533402c" (UID: "6963e146-5757-40e2-9a78-64e76533402c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:06:12.613759 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.613741    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6963e146-5757-40e2-9a78-64e76533402c-kube-api-access-gtl9b" (OuterVolumeSpecName: "kube-api-access-gtl9b") pod "6963e146-5757-40e2-9a78-64e76533402c" (UID: "6963e146-5757-40e2-9a78-64e76533402c"). InnerVolumeSpecName "kube-api-access-gtl9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:06:12.614283 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.614264    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10093fb8-71d1-49b6-aaba-d538110c1ca3" (UID: "10093fb8-71d1-49b6-aaba-d538110c1ca3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:06:12.614335 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.614285    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "10093fb8-71d1-49b6-aaba-d538110c1ca3" (UID: "10093fb8-71d1-49b6-aaba-d538110c1ca3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:06:12.614335 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.614288    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10093fb8-71d1-49b6-aaba-d538110c1ca3-kube-api-access-66kzs" (OuterVolumeSpecName: "kube-api-access-66kzs") pod "10093fb8-71d1-49b6-aaba-d538110c1ca3" (UID: "10093fb8-71d1-49b6-aaba-d538110c1ca3"). InnerVolumeSpecName "kube-api-access-66kzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:06:12.713038 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.713016    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:12.713132 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.713041    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10093fb8-71d1-49b6-aaba-d538110c1ca3-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:12.713132 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.713053    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6963e146-5757-40e2-9a78-64e76533402c-plugins-config-volume\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:12.713132 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.713063    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66kzs\" (UniqueName: \"kubernetes.io/projected/10093fb8-71d1-49b6-aaba-d538110c1ca3-kube-api-access-66kzs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:12.713132 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.713072    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtl9b\" (UniqueName: \"kubernetes.io/projected/6963e146-5757-40e2-9a78-64e76533402c-kube-api-access-gtl9b\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:12.713132 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.713081    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10093fb8-71d1-49b6-aaba-d538110c1ca3-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:12.841751 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:12.841692    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:13.308568 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.308534    2562 generic.go:356] "Generic (PLEG): container finished" podID="6963e146-5757-40e2-9a78-64e76533402c" containerID="f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095" exitCode=0
Sep 11 00:06:13.308999 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.308580    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"
Sep 11 00:06:13.308999 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.308619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl" event={"ID":"6963e146-5757-40e2-9a78-64e76533402c","Type":"ContainerDied","Data":"f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095"}
Sep 11 00:06:13.308999 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.308675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl" event={"ID":"6963e146-5757-40e2-9a78-64e76533402c","Type":"ContainerDied","Data":"0cb0b9a8c101594c096d313f6f00df8fc8b2293861d2210ace1d866183cbf405"}
Sep 11 00:06:13.308999 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.308701    2562 scope.go:117] "RemoveContainer" containerID="f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095"
Sep 11 00:06:13.317596 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.317572    2562 generic.go:356] "Generic (PLEG): container finished" podID="10093fb8-71d1-49b6-aaba-d538110c1ca3" containerID="993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65" exitCode=0
Sep 11 00:06:13.317722 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.317619    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"
Sep 11 00:06:13.317722 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.317620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw" event={"ID":"10093fb8-71d1-49b6-aaba-d538110c1ca3","Type":"ContainerDied","Data":"993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65"}
Sep 11 00:06:13.317827 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.317799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw" event={"ID":"10093fb8-71d1-49b6-aaba-d538110c1ca3","Type":"ContainerDied","Data":"4d5f2920ded81a0393664c575fb92aef757b00eea3b63c722522d5d748a69f08"}
Sep 11 00:06:13.324808 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.324791    2562 scope.go:117] "RemoveContainer" containerID="f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095"
Sep 11 00:06:13.325141 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:06:13.325124    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095\": container with ID starting with f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095 not found: ID does not exist" containerID="f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095"
Sep 11 00:06:13.325228 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.325150    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095"} err="failed to get container status \"f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095\": rpc error: code = NotFound desc = could not find container \"f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095\": container with ID starting with f83773c37277c8b520f3a79eff0419c92ad44652bfdd9c6bb3172002592d6095 not found: ID does not exist"
Sep 11 00:06:13.325228 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.325172    2562 scope.go:117] "RemoveContainer" containerID="993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65"
Sep 11 00:06:13.325870 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.325853    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"]
Sep 11 00:06:13.328728 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.328705    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4094ce-54b4d4f674-m8mpl"]
Sep 11 00:06:13.333089 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.333073    2562 scope.go:117] "RemoveContainer" containerID="993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65"
Sep 11 00:06:13.333343 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:06:13.333327    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65\": container with ID starting with 993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65 not found: ID does not exist" containerID="993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65"
Sep 11 00:06:13.333410 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.333355    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65"} err="failed to get container status \"993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65\": rpc error: code = NotFound desc = could not find container \"993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65\": container with ID starting with 993483d03f1e88c81c0c6f7a1c736b4c8c1cfdd918f2bdf75b0267aa9fe77b65 not found: ID does not exist"
Sep 11 00:06:13.339016 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.338988    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"]
Sep 11 00:06:13.342795 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:13.342773    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4094ce-86976f9997-jgzmw"]
Sep 11 00:06:14.655817 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:14.655789    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10093fb8-71d1-49b6-aaba-d538110c1ca3" path="/var/lib/kubelet/pods/10093fb8-71d1-49b6-aaba-d538110c1ca3/volumes"
Sep 11 00:06:14.656188 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:14.656143    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6963e146-5757-40e2-9a78-64e76533402c" path="/var/lib/kubelet/pods/6963e146-5757-40e2-9a78-64e76533402c/volumes"
Sep 11 00:06:14.731254 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:14.731224    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:19.204200 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:19.204165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:19.273739 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:19.273707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:19.464288 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:19.464185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:06:22.007766 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.007686    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-t9bkz"]
Sep 11 00:06:22.008181 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.007942    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz" podUID="c5aa089a-7620-4bb2-80fb-433c6f4a48f9" containerName="manager" containerID="cri-o://9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53" gracePeriod=30
Sep 11 00:06:22.254678 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.254655    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 11 00:06:22.366626 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.366551    2562 generic.go:356] "Generic (PLEG): container finished" podID="c5aa089a-7620-4bb2-80fb-433c6f4a48f9" containerID="9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53" exitCode=0
Sep 11 00:06:22.366626 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.366597    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz"
Sep 11 00:06:22.366799 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.366642    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz" event={"ID":"c5aa089a-7620-4bb2-80fb-433c6f4a48f9","Type":"ContainerDied","Data":"9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53"}
Sep 11 00:06:22.366799 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.366698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-t9bkz" event={"ID":"c5aa089a-7620-4bb2-80fb-433c6f4a48f9","Type":"ContainerDied","Data":"785291be5675a018071a254da286aa3499f8709f7c6572028856b6a418ce47ef"}
Sep 11 00:06:22.366799 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.366726    2562 scope.go:117] "RemoveContainer" containerID="9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53"
Sep 11 00:06:22.375454 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.375437    2562 scope.go:117] "RemoveContainer" containerID="9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53"
Sep 11 00:06:22.375699 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:06:22.375683    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53\": container with ID starting with 9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53 not found: ID does not exist" containerID="9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53"
Sep 11 00:06:22.375749 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.375710    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53"} err="failed to get container status \"9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53\": rpc error: code = NotFound desc = could not find container \"9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53\": container with ID starting with 9854550569b70b1d4f02bf9e236cfe595387ae20bd4f4487da0a0aab28234f53 not found: ID does not exist"
Sep 11 00:06:22.395052 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.395031    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-metrics-certs\") pod \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") "
Sep 11 00:06:22.395149 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.395061    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-webhook-cert\") pod \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") "
Sep 11 00:06:22.395245 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.395226    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9mj9\" (UniqueName: \"kubernetes.io/projected/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-kube-api-access-v9mj9\") pod \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\" (UID: \"c5aa089a-7620-4bb2-80fb-433c6f4a48f9\") "
Sep 11 00:06:22.397086 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.397056    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c5aa089a-7620-4bb2-80fb-433c6f4a48f9" (UID: "c5aa089a-7620-4bb2-80fb-433c6f4a48f9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:06:22.397190 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.397126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5aa089a-7620-4bb2-80fb-433c6f4a48f9" (UID: "c5aa089a-7620-4bb2-80fb-433c6f4a48f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:06:22.397190 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.397156    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-kube-api-access-v9mj9" (OuterVolumeSpecName: "kube-api-access-v9mj9") pod "c5aa089a-7620-4bb2-80fb-433c6f4a48f9" (UID: "c5aa089a-7620-4bb2-80fb-433c6f4a48f9"). InnerVolumeSpecName "kube-api-access-v9mj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:06:22.496358 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.496325    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9mj9\" (UniqueName: \"kubernetes.io/projected/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-kube-api-access-v9mj9\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:22.496358 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.496352    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:22.496358 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.496361    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5aa089a-7620-4bb2-80fb-433c6f4a48f9-webhook-cert\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:06:22.682173 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.682147    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-t9bkz"]
Sep 11 00:06:22.685184 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:22.685160    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-t9bkz"]
Sep 11 00:06:24.654679 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:24.654648    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5aa089a-7620-4bb2-80fb-433c6f4a48f9" path="/var/lib/kubelet/pods/c5aa089a-7620-4bb2-80fb-433c6f4a48f9/volumes"
Sep 11 00:06:28.690017 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:28.689979    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 11 00:06:28.697028 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:06:28.697006    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 11 00:07:23.481109 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481074    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-xcx9z"]
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481413    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418cd2f8-323f-4ef2-9726-294257dd3374" containerName="payload-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481426    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="418cd2f8-323f-4ef2-9726-294257dd3374" containerName="payload-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481443    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a837e4-399d-4472-9bd4-6e82b7b5c0a3" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481451    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a837e4-399d-4472-9bd4-6e82b7b5c0a3" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481463    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73df75ed-2097-49c5-a4f8-b13f3a124ce0" containerName="payload-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481468    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="73df75ed-2097-49c5-a4f8-b13f3a124ce0" containerName="payload-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481475    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00daa5cc-4569-4b43-91f1-45f03df24a49" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481480    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00daa5cc-4569-4b43-91f1-45f03df24a49" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481488    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6963e146-5757-40e2-9a78-64e76533402c" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481494    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6963e146-5757-40e2-9a78-64e76533402c" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481503    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230b78d4-0697-4607-857c-3530c6cfd281" containerName="payload-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481509    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="230b78d4-0697-4607-857c-3530c6cfd281" containerName="payload-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481518    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10093fb8-71d1-49b6-aaba-d538110c1ca3" containerName="maas-api"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481526    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="10093fb8-71d1-49b6-aaba-d538110c1ca3" containerName="maas-api"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481532    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2536378c-aa04-467d-97e3-feb97db9d3e0" containerName="maas-api"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481537    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2536378c-aa04-467d-97e3-feb97db9d3e0" containerName="maas-api"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481546    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5aa089a-7620-4bb2-80fb-433c6f4a48f9" containerName="manager"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481550    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5aa089a-7620-4bb2-80fb-433c6f4a48f9" containerName="manager"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481558    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95599a9-142f-434c-bec9-473ee207a58e" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481563    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95599a9-142f-434c-bec9-473ee207a58e" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481620    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a837e4-399d-4472-9bd4-6e82b7b5c0a3" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481630    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6963e146-5757-40e2-9a78-64e76533402c" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481636    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="10093fb8-71d1-49b6-aaba-d538110c1ca3" containerName="maas-api"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481644    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95599a9-142f-434c-bec9-473ee207a58e" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481649    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5aa089a-7620-4bb2-80fb-433c6f4a48f9" containerName="manager"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481654    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="73df75ed-2097-49c5-a4f8-b13f3a124ce0" containerName="payload-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481661    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2536378c-aa04-467d-97e3-feb97db9d3e0" containerName="maas-api"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481665    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="418cd2f8-323f-4ef2-9726-294257dd3374" containerName="payload-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481675    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00daa5cc-4569-4b43-91f1-45f03df24a49" containerName="payload-pre-processing"
Sep 11 00:07:23.483383 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.481683    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="230b78d4-0697-4607-857c-3530c6cfd281" containerName="payload-processing"
Sep 11 00:07:23.484490 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.484470    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.487938 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.487901    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 00:07:23.488063 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.487943    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 00:07:23.488063 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.487998    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mmgzc\""
Sep 11 00:07:23.488063 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.488041    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 00:07:23.494455 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.494431    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-xcx9z"]
Sep 11 00:07:23.573171 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.573142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcrl\" (UniqueName: \"kubernetes.io/projected/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-kube-api-access-2pcrl\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.573302 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.573196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-metrics-certs\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.573302 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.573279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-webhook-cert\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.673823 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.673793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-metrics-certs\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.673988 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.673836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-webhook-cert\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.673988 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.673883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcrl\" (UniqueName: \"kubernetes.io/projected/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-kube-api-access-2pcrl\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.676199 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.676178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-webhook-cert\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.676310 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.676292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-metrics-certs\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.685445 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.685422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcrl\" (UniqueName: \"kubernetes.io/projected/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-kube-api-access-2pcrl\") pod \"maas-controller-54dc99dcdb-xcx9z\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") " pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:23.795950 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:23.795860    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:24.066509 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:24.066487    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-xcx9z"]
Sep 11 00:07:24.068973 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:07:24.068922    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd37e9eb_3851_4e1c_ad22_cd279ffc4f5b.slice/crio-56a07cc1ae1d78e4de4d79d781368b798fc3d5fd53ce7ba7d9a594ee9c505240 WatchSource:0}: Error finding container 56a07cc1ae1d78e4de4d79d781368b798fc3d5fd53ce7ba7d9a594ee9c505240: Status 404 returned error can't find the container with id 56a07cc1ae1d78e4de4d79d781368b798fc3d5fd53ce7ba7d9a594ee9c505240
Sep 11 00:07:24.070292 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:24.070275    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 00:07:24.660805 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:24.660766    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z" event={"ID":"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b","Type":"ContainerStarted","Data":"56a07cc1ae1d78e4de4d79d781368b798fc3d5fd53ce7ba7d9a594ee9c505240"}
Sep 11 00:07:25.657685 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:25.657630    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z" event={"ID":"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b","Type":"ContainerStarted","Data":"8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338"}
Sep 11 00:07:25.657880 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:25.657734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:25.675876 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:25.675833    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z" podStartSLOduration=1.9049322850000001 podStartE2EDuration="2.67581996s" totalImagesPullingTime="770.887675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:07:25.675135423 +0000 UTC m=+1857.506162956" watchObservedRunningTime="2026-09-11 00:07:25.67581996 +0000 UTC m=+1857.506847492"
Sep 11 00:07:35.824343 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:35.824310    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:36.665122 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:36.665093    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:37.570543 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:37.570512    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:45.862030 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:45.862002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:45.961891 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:45.961862    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-xcx9z"]
Sep 11 00:07:45.962075 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:45.962051    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z" podUID="fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" containerName="manager" containerID="cri-o://8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338" gracePeriod=30
Sep 11 00:07:46.205152 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.205129    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:46.354164 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.354128    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-webhook-cert\") pod \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") "
Sep 11 00:07:46.354164 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.354169    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-metrics-certs\") pod \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") "
Sep 11 00:07:46.354367 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.354265    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pcrl\" (UniqueName: \"kubernetes.io/projected/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-kube-api-access-2pcrl\") pod \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\" (UID: \"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b\") "
Sep 11 00:07:46.356302 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.356266    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" (UID: "fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:07:46.356412 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.356318    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" (UID: "fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:07:46.356412 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.356328    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-kube-api-access-2pcrl" (OuterVolumeSpecName: "kube-api-access-2pcrl") pod "fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" (UID: "fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b"). InnerVolumeSpecName "kube-api-access-2pcrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:07:46.455080 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.455054    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-webhook-cert\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:07:46.455080 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.455081    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:07:46.455235 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.455092    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pcrl\" (UniqueName: \"kubernetes.io/projected/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b-kube-api-access-2pcrl\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:07:46.747666 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.747601    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" containerID="8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338" exitCode=0
Sep 11 00:07:46.747666 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.747646    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z"
Sep 11 00:07:46.747826 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.747686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z" event={"ID":"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b","Type":"ContainerDied","Data":"8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338"}
Sep 11 00:07:46.747826 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.747742    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-xcx9z" event={"ID":"fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b","Type":"ContainerDied","Data":"56a07cc1ae1d78e4de4d79d781368b798fc3d5fd53ce7ba7d9a594ee9c505240"}
Sep 11 00:07:46.747826 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.747769    2562 scope.go:117] "RemoveContainer" containerID="8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338"
Sep 11 00:07:46.756069 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.756051    2562 scope.go:117] "RemoveContainer" containerID="8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338"
Sep 11 00:07:46.756325 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:07:46.756303    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338\": container with ID starting with 8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338 not found: ID does not exist" containerID="8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338"
Sep 11 00:07:46.756418 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.756330    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338"} err="failed to get container status \"8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338\": rpc error: code = NotFound desc = could not find container \"8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338\": container with ID starting with 8a5f19dee660219ab5db00ccfd627a832c8f14e749f42ec1125b5755da4ff338 not found: ID does not exist"
Sep 11 00:07:46.763827 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.763800    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-xcx9z"]
Sep 11 00:07:46.766187 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:46.766166    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-xcx9z"]
Sep 11 00:07:47.405546 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.405508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:07:47.513747 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.513708    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-q9lmw"]
Sep 11 00:07:47.514136 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.514118    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" containerName="manager"
Sep 11 00:07:47.514238 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.514138    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" containerName="manager"
Sep 11 00:07:47.514238 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.514221    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" containerName="manager"
Sep 11 00:07:47.518511 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.518490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.522040 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.522016    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 00:07:47.522040 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.522028    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mmgzc\""
Sep 11 00:07:47.522209 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.522054    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 00:07:47.522209 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.522055    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 00:07:47.524811 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.524790    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-q9lmw"]
Sep 11 00:07:47.663805 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.663720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9m4v\" (UniqueName: \"kubernetes.io/projected/377d15fe-37b0-4eea-ab27-1e49bc897971-kube-api-access-s9m4v\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.663805 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.663765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/377d15fe-37b0-4eea-ab27-1e49bc897971-metrics-certs\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.664011 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.663817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/377d15fe-37b0-4eea-ab27-1e49bc897971-webhook-cert\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.765219 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.765186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9m4v\" (UniqueName: \"kubernetes.io/projected/377d15fe-37b0-4eea-ab27-1e49bc897971-kube-api-access-s9m4v\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.765396 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.765326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/377d15fe-37b0-4eea-ab27-1e49bc897971-metrics-certs\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.765474 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.765397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/377d15fe-37b0-4eea-ab27-1e49bc897971-webhook-cert\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.767882 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.767859    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/377d15fe-37b0-4eea-ab27-1e49bc897971-webhook-cert\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.768015 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.767901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/377d15fe-37b0-4eea-ab27-1e49bc897971-metrics-certs\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.772472 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.772451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9m4v\" (UniqueName: \"kubernetes.io/projected/377d15fe-37b0-4eea-ab27-1e49bc897971-kube-api-access-s9m4v\") pod \"maas-controller-54dc99dcdb-q9lmw\" (UID: \"377d15fe-37b0-4eea-ab27-1e49bc897971\") " pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:47.829442 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:47.829422    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:48.099003 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:48.098980    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54dc99dcdb-q9lmw"]
Sep 11 00:07:48.101240 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:07:48.101215    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377d15fe_37b0_4eea_ab27_1e49bc897971.slice/crio-6bc2221f86e9a24226992515d9021b9138eb35d9d112e88a7274c12bbdaeca69 WatchSource:0}: Error finding container 6bc2221f86e9a24226992515d9021b9138eb35d9d112e88a7274c12bbdaeca69: Status 404 returned error can't find the container with id 6bc2221f86e9a24226992515d9021b9138eb35d9d112e88a7274c12bbdaeca69
Sep 11 00:07:48.655683 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:48.655651    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b" path="/var/lib/kubelet/pods/fd37e9eb-3851-4e1c-ad22-cd279ffc4f5b/volumes"
Sep 11 00:07:48.763019 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:48.762966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-q9lmw" event={"ID":"377d15fe-37b0-4eea-ab27-1e49bc897971","Type":"ContainerStarted","Data":"6bc2221f86e9a24226992515d9021b9138eb35d9d112e88a7274c12bbdaeca69"}
Sep 11 00:07:49.769572 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:49.769535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54dc99dcdb-q9lmw" event={"ID":"377d15fe-37b0-4eea-ab27-1e49bc897971","Type":"ContainerStarted","Data":"227b5892ac0a17a3240302d071a82e60cfb2cfa8d98884ffd544226172208ec1"}
Sep 11 00:07:49.769957 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:49.769666    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:07:49.793200 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:07:49.793156    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54dc99dcdb-q9lmw" podStartSLOduration=1.850519871 podStartE2EDuration="2.79314118s" totalImagesPullingTime="942.621309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:07:49.791117138 +0000 UTC m=+1881.622144671" watchObservedRunningTime="2026-09-11 00:07:49.79314118 +0000 UTC m=+1881.624168721"
Sep 11 00:08:00.776664 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:08:00.776588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54dc99dcdb-q9lmw"
Sep 11 00:09:24.530973 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:09:24.530874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:10:17.573510 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:17.573460    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:10:34.725308 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.725270    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"]
Sep 11 00:10:34.729167 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.729150    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.731902 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.731882    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0ee0968a-kserve-self-signed-certs\""
Sep 11 00:10:34.736164 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.736146    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"]
Sep 11 00:10:34.750453 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.750422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:10:34.756414 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.756390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tmp-dir\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.756510 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.756430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv2s7\" (UniqueName: \"kubernetes.io/projected/f0ffdd5e-efa3-49d6-9b08-561a076d4832-kube-api-access-jv2s7\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.756510 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.756464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-model-cache\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.756510 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.756490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-home\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.756510 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.756506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-dshm\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.756654 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.756604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tls-certs\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.857526 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.857494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tmp-dir\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.857680 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.857541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv2s7\" (UniqueName: \"kubernetes.io/projected/f0ffdd5e-efa3-49d6-9b08-561a076d4832-kube-api-access-jv2s7\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.857680 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.857594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-model-cache\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.857680 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.857638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-home\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.857836 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.857735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-dshm\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.857836 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.857795    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tls-certs\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.857998 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.857975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tmp-dir\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.857998 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.857989    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-model-cache\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.858119 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.858073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-home\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.859841 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.859816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-dshm\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.860348 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.860326    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tls-certs\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:34.874124 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:34.874096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv2s7\" (UniqueName: \"kubernetes.io/projected/f0ffdd5e-efa3-49d6-9b08-561a076d4832-kube-api-access-jv2s7\") pod \"e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") " pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:35.038256 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:35.038180    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:35.307786 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:35.307763    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"]
Sep 11 00:10:35.309694 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:10:35.309667    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0ffdd5e_efa3_49d6_9b08_561a076d4832.slice/crio-4f113ddca865d4ccebf1e262fb6323daec05046a8d95bfb4208bb81b279ff8e1 WatchSource:0}: Error finding container 4f113ddca865d4ccebf1e262fb6323daec05046a8d95bfb4208bb81b279ff8e1: Status 404 returned error can't find the container with id 4f113ddca865d4ccebf1e262fb6323daec05046a8d95bfb4208bb81b279ff8e1
Sep 11 00:10:35.529117 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:35.529081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4" event={"ID":"f0ffdd5e-efa3-49d6-9b08-561a076d4832","Type":"ContainerStarted","Data":"76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca"}
Sep 11 00:10:35.529117 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:35.529126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4" event={"ID":"f0ffdd5e-efa3-49d6-9b08-561a076d4832","Type":"ContainerStarted","Data":"4f113ddca865d4ccebf1e262fb6323daec05046a8d95bfb4208bb81b279ff8e1"}
Sep 11 00:10:35.547078 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:35.547031    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4" podStartSLOduration=1.547016656 podStartE2EDuration="1.547016656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:10:35.545205186 +0000 UTC m=+2047.376232720" watchObservedRunningTime="2026-09-11 00:10:35.547016656 +0000 UTC m=+2047.378044211"
Sep 11 00:10:41.420612 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.420573    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"]
Sep 11 00:10:41.424497 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.424470    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.426937 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.426901    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0ee0968a-kserve-self-signed-certs\""
Sep 11 00:10:41.434327 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.434300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"]
Sep 11 00:10:41.459410 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.459385    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:10:41.513021 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.512999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-home\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.513130 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.513046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-tmp-dir\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.513130 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.513066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrxm\" (UniqueName: \"kubernetes.io/projected/f34a9d1f-ca99-45d3-90de-f43682054610-kube-api-access-xgrxm\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.513202 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.513123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f34a9d1f-ca99-45d3-90de-f43682054610-tls-certs\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.513202 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.513181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-model-cache\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.513268 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.513204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-dshm\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614352 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f34a9d1f-ca99-45d3-90de-f43682054610-tls-certs\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614457 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-model-cache\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614457 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614392    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-dshm\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614457 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-home\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614599 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614459    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-tmp-dir\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614599 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrxm\" (UniqueName: \"kubernetes.io/projected/f34a9d1f-ca99-45d3-90de-f43682054610-kube-api-access-xgrxm\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614809 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-model-cache\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614877 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-tmp-dir\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.614954 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.614913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-home\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.616679 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.616662    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-dshm\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.617115 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.617093    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f34a9d1f-ca99-45d3-90de-f43682054610-tls-certs\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.623992 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.623972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrxm\" (UniqueName: \"kubernetes.io/projected/f34a9d1f-ca99-45d3-90de-f43682054610-kube-api-access-xgrxm\") pod \"e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") " pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:41.734277 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:41.734256    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:42.007045 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:42.006989    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"]
Sep 11 00:10:42.009209 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:10:42.009181    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf34a9d1f_ca99_45d3_90de_f43682054610.slice/crio-c0db6ea34af7a5f5c82a04a1b06c2cf6799dd6a1f4b17d4e85f34fcdcf9ead81 WatchSource:0}: Error finding container c0db6ea34af7a5f5c82a04a1b06c2cf6799dd6a1f4b17d4e85f34fcdcf9ead81: Status 404 returned error can't find the container with id c0db6ea34af7a5f5c82a04a1b06c2cf6799dd6a1f4b17d4e85f34fcdcf9ead81
Sep 11 00:10:42.561726 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:42.561690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64" event={"ID":"f34a9d1f-ca99-45d3-90de-f43682054610","Type":"ContainerStarted","Data":"7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece"}
Sep 11 00:10:42.561726 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:42.561733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64" event={"ID":"f34a9d1f-ca99-45d3-90de-f43682054610","Type":"ContainerStarted","Data":"c0db6ea34af7a5f5c82a04a1b06c2cf6799dd6a1f4b17d4e85f34fcdcf9ead81"}
Sep 11 00:10:42.584706 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:42.584653    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64" podStartSLOduration=1.58463725 podStartE2EDuration="1.58463725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:10:42.582799611 +0000 UTC m=+2054.413827168" watchObservedRunningTime="2026-09-11 00:10:42.58463725 +0000 UTC m=+2054.415664783"
Sep 11 00:10:45.038734 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:45.038700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:45.039148 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:45.038768    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:45.050912 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:45.050890    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:45.585308 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:45.585274    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:10:51.735037 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:51.734963    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:51.735037 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:51.735013    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:51.747648 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:51.747621    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:52.615921 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:52.615885    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:10:58.585045 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:58.585012    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"]
Sep 11 00:10:58.611699 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:58.611671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:10:59.690668 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:59.690631    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"]
Sep 11 00:10:59.711228 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:10:59.711202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:11:13.604103 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.604047    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4" podUID="f0ffdd5e-efa3-49d6-9b08-561a076d4832" containerName="main" containerID="cri-o://76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca" gracePeriod=45
Sep 11 00:11:13.851267 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.851237    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:11:13.981855 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.981827    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv2s7\" (UniqueName: \"kubernetes.io/projected/f0ffdd5e-efa3-49d6-9b08-561a076d4832-kube-api-access-jv2s7\") pod \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") "
Sep 11 00:11:13.982067 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.981892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-home\") pod \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") "
Sep 11 00:11:13.982067 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.981911    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-dshm\") pod \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") "
Sep 11 00:11:13.982067 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.981973    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tmp-dir\") pod \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") "
Sep 11 00:11:13.982067 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.982004    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-model-cache\") pod \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") "
Sep 11 00:11:13.982067 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.982054    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tls-certs\") pod \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\" (UID: \"f0ffdd5e-efa3-49d6-9b08-561a076d4832\") "
Sep 11 00:11:13.982412 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.982220    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-home" (OuterVolumeSpecName: "home") pod "f0ffdd5e-efa3-49d6-9b08-561a076d4832" (UID: "f0ffdd5e-efa3-49d6-9b08-561a076d4832"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:11:13.982412 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.982255    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0ffdd5e-efa3-49d6-9b08-561a076d4832" (UID: "f0ffdd5e-efa3-49d6-9b08-561a076d4832"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:11:13.982412 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.982300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0ffdd5e-efa3-49d6-9b08-561a076d4832" (UID: "f0ffdd5e-efa3-49d6-9b08-561a076d4832"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:11:13.982412 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.982380    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:13.982562 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.982423    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:13.982562 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.982438    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:13.984172 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.984148    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-dshm" (OuterVolumeSpecName: "dshm") pod "f0ffdd5e-efa3-49d6-9b08-561a076d4832" (UID: "f0ffdd5e-efa3-49d6-9b08-561a076d4832"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:11:13.984172 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.984157    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ffdd5e-efa3-49d6-9b08-561a076d4832-kube-api-access-jv2s7" (OuterVolumeSpecName: "kube-api-access-jv2s7") pod "f0ffdd5e-efa3-49d6-9b08-561a076d4832" (UID: "f0ffdd5e-efa3-49d6-9b08-561a076d4832"). InnerVolumeSpecName "kube-api-access-jv2s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:11:13.984295 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:13.984216    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0ffdd5e-efa3-49d6-9b08-561a076d4832" (UID: "f0ffdd5e-efa3-49d6-9b08-561a076d4832"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:11:14.083322 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.083297    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv2s7\" (UniqueName: \"kubernetes.io/projected/f0ffdd5e-efa3-49d6-9b08-561a076d4832-kube-api-access-jv2s7\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:14.083322 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.083321    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ffdd5e-efa3-49d6-9b08-561a076d4832-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:14.083458 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.083332    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ffdd5e-efa3-49d6-9b08-561a076d4832-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:14.703708 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.703681    2562 generic.go:356] "Generic (PLEG): container finished" podID="f0ffdd5e-efa3-49d6-9b08-561a076d4832" containerID="76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca" exitCode=0
Sep 11 00:11:14.704069 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.703720    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"
Sep 11 00:11:14.704069 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.703763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4" event={"ID":"f0ffdd5e-efa3-49d6-9b08-561a076d4832","Type":"ContainerDied","Data":"76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca"}
Sep 11 00:11:14.704069 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.703805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4" event={"ID":"f0ffdd5e-efa3-49d6-9b08-561a076d4832","Type":"ContainerDied","Data":"4f113ddca865d4ccebf1e262fb6323daec05046a8d95bfb4208bb81b279ff8e1"}
Sep 11 00:11:14.704069 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.703823    2562 scope.go:117] "RemoveContainer" containerID="76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca"
Sep 11 00:11:14.710458 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.710425    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64" podUID="f34a9d1f-ca99-45d3-90de-f43682054610" containerName="main" containerID="cri-o://7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece" gracePeriod=45
Sep 11 00:11:14.715502 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.715479    2562 scope.go:117] "RemoveContainer" containerID="76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca"
Sep 11 00:11:14.716050 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:11:14.716025    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca\": container with ID starting with 76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca not found: ID does not exist" containerID="76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca"
Sep 11 00:11:14.716137 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.716061    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca"} err="failed to get container status \"76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca\": rpc error: code = NotFound desc = could not find container \"76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca\": container with ID starting with 76e2b13e1d58a098d47e4520fc8d295cc79aa7799c3ce8da3425f54a35b6f5ca not found: ID does not exist"
Sep 11 00:11:14.720753 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.720728    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"]
Sep 11 00:11:14.724330 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.724311    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0ee0968a-kserve-5c9646b479-84sx4"]
Sep 11 00:11:14.951861 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.951840    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:11:14.989404 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-home\") pod \"f34a9d1f-ca99-45d3-90de-f43682054610\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") "
Sep 11 00:11:14.989533 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f34a9d1f-ca99-45d3-90de-f43682054610-tls-certs\") pod \"f34a9d1f-ca99-45d3-90de-f43682054610\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") "
Sep 11 00:11:14.989533 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989456    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-model-cache\") pod \"f34a9d1f-ca99-45d3-90de-f43682054610\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") "
Sep 11 00:11:14.989533 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989487    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgrxm\" (UniqueName: \"kubernetes.io/projected/f34a9d1f-ca99-45d3-90de-f43682054610-kube-api-access-xgrxm\") pod \"f34a9d1f-ca99-45d3-90de-f43682054610\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") "
Sep 11 00:11:14.989706 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989536    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-dshm\") pod \"f34a9d1f-ca99-45d3-90de-f43682054610\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") "
Sep 11 00:11:14.989706 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-tmp-dir\") pod \"f34a9d1f-ca99-45d3-90de-f43682054610\" (UID: \"f34a9d1f-ca99-45d3-90de-f43682054610\") "
Sep 11 00:11:14.989706 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989626    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-home" (OuterVolumeSpecName: "home") pod "f34a9d1f-ca99-45d3-90de-f43682054610" (UID: "f34a9d1f-ca99-45d3-90de-f43682054610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:11:14.989867 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989770    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-model-cache" (OuterVolumeSpecName: "model-cache") pod "f34a9d1f-ca99-45d3-90de-f43682054610" (UID: "f34a9d1f-ca99-45d3-90de-f43682054610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:11:14.989920 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989864    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f34a9d1f-ca99-45d3-90de-f43682054610" (UID: "f34a9d1f-ca99-45d3-90de-f43682054610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:11:14.989920 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989899    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-home\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:14.990032 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.989946    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-model-cache\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:14.991501 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.991482    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34a9d1f-ca99-45d3-90de-f43682054610-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f34a9d1f-ca99-45d3-90de-f43682054610" (UID: "f34a9d1f-ca99-45d3-90de-f43682054610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:11:14.992026 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.992005    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-dshm" (OuterVolumeSpecName: "dshm") pod "f34a9d1f-ca99-45d3-90de-f43682054610" (UID: "f34a9d1f-ca99-45d3-90de-f43682054610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:11:14.992119 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:14.992004    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34a9d1f-ca99-45d3-90de-f43682054610-kube-api-access-xgrxm" (OuterVolumeSpecName: "kube-api-access-xgrxm") pod "f34a9d1f-ca99-45d3-90de-f43682054610" (UID: "f34a9d1f-ca99-45d3-90de-f43682054610"). InnerVolumeSpecName "kube-api-access-xgrxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:11:15.090340 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.090315    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-tmp-dir\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:15.090340 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.090336    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f34a9d1f-ca99-45d3-90de-f43682054610-tls-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:15.090340 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.090345    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgrxm\" (UniqueName: \"kubernetes.io/projected/f34a9d1f-ca99-45d3-90de-f43682054610-kube-api-access-xgrxm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:15.090512 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.090354    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f34a9d1f-ca99-45d3-90de-f43682054610-dshm\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:15.711228 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.711195    2562 generic.go:356] "Generic (PLEG): container finished" podID="f34a9d1f-ca99-45d3-90de-f43682054610" containerID="7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece" exitCode=0
Sep 11 00:11:15.711678 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.711241    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"
Sep 11 00:11:15.711678 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.711279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64" event={"ID":"f34a9d1f-ca99-45d3-90de-f43682054610","Type":"ContainerDied","Data":"7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece"}
Sep 11 00:11:15.711678 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.711325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64" event={"ID":"f34a9d1f-ca99-45d3-90de-f43682054610","Type":"ContainerDied","Data":"c0db6ea34af7a5f5c82a04a1b06c2cf6799dd6a1f4b17d4e85f34fcdcf9ead81"}
Sep 11 00:11:15.711678 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.711348    2562 scope.go:117] "RemoveContainer" containerID="7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece"
Sep 11 00:11:15.720966 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.720948    2562 scope.go:117] "RemoveContainer" containerID="7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece"
Sep 11 00:11:15.721219 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:11:15.721199    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece\": container with ID starting with 7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece not found: ID does not exist" containerID="7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece"
Sep 11 00:11:15.721290 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.721228    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece"} err="failed to get container status \"7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece\": rpc error: code = NotFound desc = could not find container \"7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece\": container with ID starting with 7bad4b6996c5cd43866ffe5d8ec38ae60251558320a88538eeb1806ceb0e1ece not found: ID does not exist"
Sep 11 00:11:15.731883 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.731860    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"]
Sep 11 00:11:15.736233 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:15.736210    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0ee0968a-kserve-c8fdb847f-szn64"]
Sep 11 00:11:16.661214 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:16.656098    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ffdd5e-efa3-49d6-9b08-561a076d4832" path="/var/lib/kubelet/pods/f0ffdd5e-efa3-49d6-9b08-561a076d4832/volumes"
Sep 11 00:11:16.661214 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:16.656681    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f34a9d1f-ca99-45d3-90de-f43682054610" path="/var/lib/kubelet/pods/f34a9d1f-ca99-45d3-90de-f43682054610/volumes"
Sep 11 00:11:17.834027 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.833985    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"]
Sep 11 00:11:17.834476 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.834331    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ffdd5e-efa3-49d6-9b08-561a076d4832" containerName="main"
Sep 11 00:11:17.834476 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.834346    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ffdd5e-efa3-49d6-9b08-561a076d4832" containerName="main"
Sep 11 00:11:17.834476 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.834360    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34a9d1f-ca99-45d3-90de-f43682054610" containerName="main"
Sep 11 00:11:17.834476 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.834366    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34a9d1f-ca99-45d3-90de-f43682054610" containerName="main"
Sep 11 00:11:17.834476 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.834443    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ffdd5e-efa3-49d6-9b08-561a076d4832" containerName="main"
Sep 11 00:11:17.834476 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.834450    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34a9d1f-ca99-45d3-90de-f43682054610" containerName="main"
Sep 11 00:11:17.841266 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.841249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:17.844060 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.844029    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9tsnw\""
Sep 11 00:11:17.845582 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.845559    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"]
Sep 11 00:11:17.910653 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.910624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75aa7a75-0a12-4046-895a-ee389bbe64ba-metrics-certs\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:17.910813 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.910666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75aa7a75-0a12-4046-895a-ee389bbe64ba-openshift-service-ca\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:17.910813 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.910708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfh8\" (UniqueName: \"kubernetes.io/projected/75aa7a75-0a12-4046-895a-ee389bbe64ba-kube-api-access-vpfh8\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:17.910813 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:17.910781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75aa7a75-0a12-4046-895a-ee389bbe64ba-maas-api-tls\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.012188 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.012158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75aa7a75-0a12-4046-895a-ee389bbe64ba-metrics-certs\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.012345 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.012190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75aa7a75-0a12-4046-895a-ee389bbe64ba-openshift-service-ca\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.012345 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.012219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfh8\" (UniqueName: \"kubernetes.io/projected/75aa7a75-0a12-4046-895a-ee389bbe64ba-kube-api-access-vpfh8\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.012345 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.012262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75aa7a75-0a12-4046-895a-ee389bbe64ba-maas-api-tls\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.012808 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.012784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75aa7a75-0a12-4046-895a-ee389bbe64ba-openshift-service-ca\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.014672 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.014650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75aa7a75-0a12-4046-895a-ee389bbe64ba-maas-api-tls\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.014756 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.014700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75aa7a75-0a12-4046-895a-ee389bbe64ba-metrics-certs\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.019642 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.019618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfh8\" (UniqueName: \"kubernetes.io/projected/75aa7a75-0a12-4046-895a-ee389bbe64ba-kube-api-access-vpfh8\") pod \"maas-api-796ffbb698-t9kmx\" (UID: \"75aa7a75-0a12-4046-895a-ee389bbe64ba\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.153647 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.153569    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:18.423347 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.423320    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"]
Sep 11 00:11:18.425381 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:11:18.425353    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75aa7a75_0a12_4046_895a_ee389bbe64ba.slice/crio-cba78059c27f3719b720f95fc60e7307f887523240e9bec211ff42d453678b9f WatchSource:0}: Error finding container cba78059c27f3719b720f95fc60e7307f887523240e9bec211ff42d453678b9f: Status 404 returned error can't find the container with id cba78059c27f3719b720f95fc60e7307f887523240e9bec211ff42d453678b9f
Sep 11 00:11:18.726311 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:18.726279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx" event={"ID":"75aa7a75-0a12-4046-895a-ee389bbe64ba","Type":"ContainerStarted","Data":"cba78059c27f3719b720f95fc60e7307f887523240e9bec211ff42d453678b9f"}
Sep 11 00:11:19.732561 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:19.732525    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx" event={"ID":"75aa7a75-0a12-4046-895a-ee389bbe64ba","Type":"ContainerStarted","Data":"3739c5d56be47e4861bafee54fe001630794a7e54dcd5f5ec2d35a784493876e"}
Sep 11 00:11:19.733050 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:19.732645    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:19.749639 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:19.749591    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx" podStartSLOduration=2.363751347 podStartE2EDuration="2.749577163s" totalImagesPullingTime="385.825816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:11:19.748712947 +0000 UTC m=+2091.579740493" watchObservedRunningTime="2026-09-11 00:11:19.749577163 +0000 UTC m=+2091.580604696"
Sep 11 00:11:25.741745 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:25.741712    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796ffbb698-t9kmx"
Sep 11 00:11:28.717419 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:28.717384    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 11 00:11:28.727254 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:28.727232    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 11 00:11:33.063646 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:33.063611    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"]
Sep 11 00:11:33.064040 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:33.063857    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78" podUID="a2292ac2-da55-4a36-81cd-ffcf33659dac" containerName="maas-api" containerID="cri-o://307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f" gracePeriod=30
Sep 11 00:11:34.302302 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.302278    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 11 00:11:34.344643 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.344607    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs\") pod \"a2292ac2-da55-4a36-81cd-ffcf33659dac\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") "
Sep 11 00:11:34.344788 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.344699    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2292ac2-da55-4a36-81cd-ffcf33659dac-openshift-service-ca\") pod \"a2292ac2-da55-4a36-81cd-ffcf33659dac\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") "
Sep 11 00:11:34.344788 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.344722    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls\") pod \"a2292ac2-da55-4a36-81cd-ffcf33659dac\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") "
Sep 11 00:11:34.344788 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.344750    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6w4p\" (UniqueName: \"kubernetes.io/projected/a2292ac2-da55-4a36-81cd-ffcf33659dac-kube-api-access-r6w4p\") pod \"a2292ac2-da55-4a36-81cd-ffcf33659dac\" (UID: \"a2292ac2-da55-4a36-81cd-ffcf33659dac\") "
Sep 11 00:11:34.345069 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.345048    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2292ac2-da55-4a36-81cd-ffcf33659dac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2292ac2-da55-4a36-81cd-ffcf33659dac" (UID: "a2292ac2-da55-4a36-81cd-ffcf33659dac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:11:34.346783 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.346752    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2292ac2-da55-4a36-81cd-ffcf33659dac-kube-api-access-r6w4p" (OuterVolumeSpecName: "kube-api-access-r6w4p") pod "a2292ac2-da55-4a36-81cd-ffcf33659dac" (UID: "a2292ac2-da55-4a36-81cd-ffcf33659dac"). InnerVolumeSpecName "kube-api-access-r6w4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:11:34.346783 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.346761    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a2292ac2-da55-4a36-81cd-ffcf33659dac" (UID: "a2292ac2-da55-4a36-81cd-ffcf33659dac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:11:34.347001 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.346862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2292ac2-da55-4a36-81cd-ffcf33659dac" (UID: "a2292ac2-da55-4a36-81cd-ffcf33659dac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:11:34.446036 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.446004    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2292ac2-da55-4a36-81cd-ffcf33659dac-openshift-service-ca\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:34.446036 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.446034    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-maas-api-tls\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:34.446205 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.446047    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6w4p\" (UniqueName: \"kubernetes.io/projected/a2292ac2-da55-4a36-81cd-ffcf33659dac-kube-api-access-r6w4p\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:34.446205 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.446058    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2292ac2-da55-4a36-81cd-ffcf33659dac-metrics-certs\") on node \"ip-10-0-137-218.ec2.internal\" DevicePath \"\""
Sep 11 00:11:34.800381 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.800314    2562 generic.go:356] "Generic (PLEG): container finished" podID="a2292ac2-da55-4a36-81cd-ffcf33659dac" containerID="307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f" exitCode=0
Sep 11 00:11:34.800381 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.800353    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"
Sep 11 00:11:34.800549 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.800394    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78" event={"ID":"a2292ac2-da55-4a36-81cd-ffcf33659dac","Type":"ContainerDied","Data":"307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f"}
Sep 11 00:11:34.800549 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.800444    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78" event={"ID":"a2292ac2-da55-4a36-81cd-ffcf33659dac","Type":"ContainerDied","Data":"43809159daad411fea5b2c65e076f87457bdf2ab9f5c614b90efc0634bf5eb6e"}
Sep 11 00:11:34.800549 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.800468    2562 scope.go:117] "RemoveContainer" containerID="307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f"
Sep 11 00:11:34.809277 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.809259    2562 scope.go:117] "RemoveContainer" containerID="307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f"
Sep 11 00:11:34.809527 ip-10-0-137-218 kubenswrapper[2562]: E0911 00:11:34.809508    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f\": container with ID starting with 307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f not found: ID does not exist" containerID="307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f"
Sep 11 00:11:34.809574 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.809535    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f"} err="failed to get container status \"307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f\": rpc error: code = NotFound desc = could not find container \"307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f\": container with ID starting with 307f05211d3d3b2fdab43ca1d140356520093e2f8b9e34882c2a0c729c95049f not found: ID does not exist"
Sep 11 00:11:34.817548 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.817525    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"]
Sep 11 00:11:34.820840 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:34.820816    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-blm78"]
Sep 11 00:11:36.655266 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:36.655236    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2292ac2-da55-4a36-81cd-ffcf33659dac" path="/var/lib/kubelet/pods/a2292ac2-da55-4a36-81cd-ffcf33659dac/volumes"
Sep 11 00:11:57.774771 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:57.774739    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54dc99dcdb-q9lmw_377d15fe-37b0-4eea-ab27-1e49bc897971/manager/0.log"
Sep 11 00:11:57.885009 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:57.884980    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-psxg2_0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec/server/0.log"
Sep 11 00:11:58.017213 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:58.017186    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fpw9m_aa6a65e9-9daf-4ed1-a14d-a2d9007b103d/manager/0.log"
Sep 11 00:11:58.136920 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:58.136844    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-gnjz8_fbc0ac29-8968-4977-9be0-dbc149d88477/copy-manifests/0.log"
Sep 11 00:11:58.144725 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:58.144696    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-gnjz8_fbc0ac29-8968-4977-9be0-dbc149d88477/manager/0.log"
Sep 11 00:11:58.877837 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:58.877807    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796ffbb698-t9kmx_75aa7a75-0a12-4046-895a-ee389bbe64ba/maas-api/0.log"
Sep 11 00:11:59.105447 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:11:59.105408    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-sr9bv_9a1599a9-3e16-4923-9482-2155fdeebff9/postgres/0.log"
Sep 11 00:12:00.649514 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:00.649475    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-wdrx7_947b5b3a-6a42-41b9-8680-c504b4299c7d/manager/0.log"
Sep 11 00:12:01.758960 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:01.758911    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5hmzr_eefba04a-fca1-4281-9427-8635b54bd72a/payload-processing/0.log"
Sep 11 00:12:01.876194 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:01.876169    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c5d97497d-trwfb_12bf9c01-3b14-47e8-97a1-cd867914835f/router/0.log"
Sep 11 00:12:02.450333 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:02.450301    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-j65m4_77b2f51e-658c-4037-8dd4-6b3e0eb2b6ed/main/0.log"
Sep 11 00:12:02.670597 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:02.670569    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fksb9b_5f647124-63d3-40d9-9634-b97bcbe915a1/main/0.log"
Sep 11 00:12:08.475598 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.475566    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rq2f8/must-gather-5684v"]
Sep 11 00:12:08.476008 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.475960    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2292ac2-da55-4a36-81cd-ffcf33659dac" containerName="maas-api"
Sep 11 00:12:08.476008 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.475975    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2292ac2-da55-4a36-81cd-ffcf33659dac" containerName="maas-api"
Sep 11 00:12:08.476085 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.476036    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2292ac2-da55-4a36-81cd-ffcf33659dac" containerName="maas-api"
Sep 11 00:12:08.479039 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.479025    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq2f8/must-gather-5684v"
Sep 11 00:12:08.482325 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.482303    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rq2f8\"/\"default-dockercfg-7cn6g\""
Sep 11 00:12:08.482627 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.482607    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rq2f8\"/\"kube-root-ca.crt\""
Sep 11 00:12:08.483124 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.483102    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rq2f8\"/\"openshift-service-ca.crt\""
Sep 11 00:12:08.491421 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.491396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq2f8/must-gather-5684v"]
Sep 11 00:12:08.496002 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.495972    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:12:08.518574 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.518550    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/382bc014-055b-433f-87c3-69c6821bf281-must-gather-output\") pod \"must-gather-5684v\" (UID: \"382bc014-055b-433f-87c3-69c6821bf281\") " pod="openshift-must-gather-rq2f8/must-gather-5684v"
Sep 11 00:12:08.518678 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.518606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnxc\" (UniqueName: \"kubernetes.io/projected/382bc014-055b-433f-87c3-69c6821bf281-kube-api-access-tnnxc\") pod \"must-gather-5684v\" (UID: \"382bc014-055b-433f-87c3-69c6821bf281\") " pod="openshift-must-gather-rq2f8/must-gather-5684v"
Sep 11 00:12:08.596128 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.596100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log" limit=1024
Sep 11 00:12:08.619278 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.619251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnxc\" (UniqueName: \"kubernetes.io/projected/382bc014-055b-433f-87c3-69c6821bf281-kube-api-access-tnnxc\") pod \"must-gather-5684v\" (UID: \"382bc014-055b-433f-87c3-69c6821bf281\") " pod="openshift-must-gather-rq2f8/must-gather-5684v"
Sep 11 00:12:08.619363 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.619315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/382bc014-055b-433f-87c3-69c6821bf281-must-gather-output\") pod \"must-gather-5684v\" (UID: \"382bc014-055b-433f-87c3-69c6821bf281\") " pod="openshift-must-gather-rq2f8/must-gather-5684v"
Sep 11 00:12:08.619576 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.619557    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/382bc014-055b-433f-87c3-69c6821bf281-must-gather-output\") pod \"must-gather-5684v\" (UID: \"382bc014-055b-433f-87c3-69c6821bf281\") " pod="openshift-must-gather-rq2f8/must-gather-5684v"
Sep 11 00:12:08.630255 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.630205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnxc\" (UniqueName: \"kubernetes.io/projected/382bc014-055b-433f-87c3-69c6821bf281-kube-api-access-tnnxc\") pod \"must-gather-5684v\" (UID: \"382bc014-055b-433f-87c3-69c6821bf281\") " pod="openshift-must-gather-rq2f8/must-gather-5684v"
Sep 11 00:12:08.787867 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:08.787843    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq2f8/must-gather-5684v"
Sep 11 00:12:09.056314 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:09.056287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq2f8/must-gather-5684v"]
Sep 11 00:12:09.059258 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:12:09.059223    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod382bc014_055b_433f_87c3_69c6821bf281.slice/crio-d52f3af1fbafc36e158260fa2ff4811dfc399f71bedcdc6808cc492521f47e98 WatchSource:0}: Error finding container d52f3af1fbafc36e158260fa2ff4811dfc399f71bedcdc6808cc492521f47e98: Status 404 returned error can't find the container with id d52f3af1fbafc36e158260fa2ff4811dfc399f71bedcdc6808cc492521f47e98
Sep 11 00:12:09.955732 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:09.955703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/must-gather-5684v" event={"ID":"382bc014-055b-433f-87c3-69c6821bf281","Type":"ContainerStarted","Data":"d52f3af1fbafc36e158260fa2ff4811dfc399f71bedcdc6808cc492521f47e98"}
Sep 11 00:12:10.965334 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:10.965299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/must-gather-5684v" event={"ID":"382bc014-055b-433f-87c3-69c6821bf281","Type":"ContainerStarted","Data":"3a07bb8b1fb4356409bd5dd88655e885d6a0c611e925c29e4340dae72373f88d"}
Sep 11 00:12:10.965756 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:10.965351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/must-gather-5684v" event={"ID":"382bc014-055b-433f-87c3-69c6821bf281","Type":"ContainerStarted","Data":"6c20a559b214964f0da751f51a0e9a61a563fe2faaaaa745ab6b9946d5446b24"}
Sep 11 00:12:10.982015 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:10.981952    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rq2f8/must-gather-5684v" podStartSLOduration=2.095443414 podStartE2EDuration="2.981911605s" totalImagesPullingTime="886.468191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:12:10.979617324 +0000 UTC m=+2142.810644875" watchObservedRunningTime="2026-09-11 00:12:10.981911605 +0000 UTC m=+2142.812939142"
Sep 11 00:12:11.574571 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:11.574531    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qmbgq_3c15c11b-b1b2-4c3b-a4c8-edc70777af3d/global-pull-secret-syncer/0.log"
Sep 11 00:12:11.635064 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:11.635023    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7br8t_d4962449-bcd3-4306-b62a-0de981ea1bdc/konnectivity-agent/0.log"
Sep 11 00:12:11.758345 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:11.758291    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-218.ec2.internal_4e5c14c36611ccf2603ecc0ec2faf0be/haproxy/0.log"
Sep 11 00:12:16.585804 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:16.585768    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-wdrx7_947b5b3a-6a42-41b9-8680-c504b4299c7d/manager/0.log"
Sep 11 00:12:18.183951 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:18.181228    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-n86rk_661b8c3f-b66f-4989-ae4d-2ceb5e3b831f/cluster-monitoring-operator/0.log"
Sep 11 00:12:18.515993 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:18.515903    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgrnb_0e91ccb5-538d-4429-9afa-501c7d9b8f6d/node-exporter/0.log"
Sep 11 00:12:18.536157 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:18.536125    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgrnb_0e91ccb5-538d-4429-9afa-501c7d9b8f6d/kube-rbac-proxy/0.log"
Sep 11 00:12:18.555782 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:18.555754    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgrnb_0e91ccb5-538d-4429-9afa-501c7d9b8f6d/init-textfile/0.log"
Sep 11 00:12:21.038356 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.038266    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/1.log"
Sep 11 00:12:21.048521 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.048471    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-gw9pf_43537109-e795-4d12-9269-b5578783dc5e/console-operator/2.log"
Sep 11 00:12:21.393002 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.392065    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"]
Sep 11 00:12:21.400643 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.400617    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.403354 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.403325    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"]
Sep 11 00:12:21.445524 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.445470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-proc\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.445698 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.445583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-sys\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.445698 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.445614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrl7q\" (UniqueName: \"kubernetes.io/projected/deb43bac-8665-408e-aba1-412e747b371e-kube-api-access-rrl7q\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.445698 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.445662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-lib-modules\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.445698 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.445688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-podres\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.546955 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.546903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-sys\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.547372 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.547347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrl7q\" (UniqueName: \"kubernetes.io/projected/deb43bac-8665-408e-aba1-412e747b371e-kube-api-access-rrl7q\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.547995 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.547973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-lib-modules\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.548357 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.548335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-podres\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.548497 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.548474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-proc\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.548604 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.547268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-sys\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.548604 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.548275    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-lib-modules\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.548604 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.548587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-podres\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.548763 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.548649    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/deb43bac-8665-408e-aba1-412e747b371e-proc\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.557897 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.557876    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrl7q\" (UniqueName: \"kubernetes.io/projected/deb43bac-8665-408e-aba1-412e747b371e-kube-api-access-rrl7q\") pod \"perf-node-gather-daemonset-788cp\" (UID: \"deb43bac-8665-408e-aba1-412e747b371e\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:21.718909 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:21.718875    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:22.024119 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:22.023947    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"]
Sep 11 00:12:22.026562 ip-10-0-137-218 kubenswrapper[2562]: W0911 00:12:22.026531    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddeb43bac_8665_408e_aba1_412e747b371e.slice/crio-b1f3b1632d7f4c2f8eb602bbc29bd70344c62be3583c85a8989533cc47908ab4 WatchSource:0}: Error finding container b1f3b1632d7f4c2f8eb602bbc29bd70344c62be3583c85a8989533cc47908ab4: Status 404 returned error can't find the container with id b1f3b1632d7f4c2f8eb602bbc29bd70344c62be3583c85a8989533cc47908ab4
Sep 11 00:12:22.049266 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:22.049227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp" event={"ID":"deb43bac-8665-408e-aba1-412e747b371e","Type":"ContainerStarted","Data":"b1f3b1632d7f4c2f8eb602bbc29bd70344c62be3583c85a8989533cc47908ab4"}
Sep 11 00:12:22.265971 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:22.265915    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-r9bdr_a93fdd16-d399-4023-810e-d611f8f7aee0/volume-data-source-validator/0.log"
Sep 11 00:12:23.063071 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:23.063030    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp" event={"ID":"deb43bac-8665-408e-aba1-412e747b371e","Type":"ContainerStarted","Data":"4e13fd4ca88019be309cddf3dabab80b152ed3616be2d45255ef1815a41e8c8e"}
Sep 11 00:12:23.063904 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:23.063864    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:23.082096 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:23.082038    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp" podStartSLOduration=2.082016684 podStartE2EDuration="2.082016684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:12:23.080608388 +0000 UTC m=+2154.911635997" watchObservedRunningTime="2026-09-11 00:12:23.082016684 +0000 UTC m=+2154.913044218"
Sep 11 00:12:23.410300 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:23.410223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2jz8n_915d900b-a1a6-428b-ac3a-0641215f0233/dns/0.log"
Sep 11 00:12:23.428422 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:23.428389    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2jz8n_915d900b-a1a6-428b-ac3a-0641215f0233/kube-rbac-proxy/0.log"
Sep 11 00:12:23.552454 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:23.552350    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gvg49_1e324660-2423-4acb-becf-9a78be67198a/dns-node-resolver/0.log"
Sep 11 00:12:24.047808 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:24.047776    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29818080-ksv5c_733adc40-b33d-4910-bed7-fe7cd1bef11d/image-pruner/0.log"
Sep 11 00:12:24.099870 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:24.099845    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2q2tj_6f736cd6-ad74-4550-a2a8-82f59ee9d98f/node-ca/0.log"
Sep 11 00:12:25.614639 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:25.614605    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5hmzr_eefba04a-fca1-4281-9427-8635b54bd72a/payload-processing/0.log"
Sep 11 00:12:25.637442 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:25.637412    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c5d97497d-trwfb_12bf9c01-3b14-47e8-97a1-cd867914835f/router/0.log"
Sep 11 00:12:26.523780 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:26.523748    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jvt2x_156442c4-f848-4a58-899f-26ea730df356/serve-healthcheck-canary/0.log"
Sep 11 00:12:26.959421 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:26.959389    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-wdgcw_17c84d6c-7d15-4b25-be27-78c331ceb2ae/insights-operator/0.log"
Sep 11 00:12:26.960438 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:26.960397    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-wdgcw_17c84d6c-7d15-4b25-be27-78c331ceb2ae/insights-operator/1.log"
Sep 11 00:12:27.037006 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:27.036972    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v9r7s_13e04c4f-4b29-4167-af11-0f11b8810fae/kube-rbac-proxy/0.log"
Sep 11 00:12:27.054133 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:27.054108    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v9r7s_13e04c4f-4b29-4167-af11-0f11b8810fae/exporter/0.log"
Sep 11 00:12:27.072918 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:27.072889    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v9r7s_13e04c4f-4b29-4167-af11-0f11b8810fae/extractor/0.log"
Sep 11 00:12:29.077968 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:29.077916    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-788cp"
Sep 11 00:12:29.567558 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:29.567519    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54dc99dcdb-q9lmw_377d15fe-37b0-4eea-ab27-1e49bc897971/manager/0.log"
Sep 11 00:12:29.588370 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:29.588341    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-psxg2_0d6950ff-78d3-461f-a9fe-0f2e7ec3c1ec/server/0.log"
Sep 11 00:12:29.733776 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:29.733740    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fpw9m_aa6a65e9-9daf-4ed1-a14d-a2d9007b103d/manager/0.log"
Sep 11 00:12:29.756210 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:29.756163    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-gnjz8_fbc0ac29-8968-4977-9be0-dbc149d88477/manager/0.log"
Sep 11 00:12:29.774262 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:29.774237    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-gnjz8_fbc0ac29-8968-4977-9be0-dbc149d88477/copy-manifests/0.log"
Sep 11 00:12:31.384591 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:31.384505    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66bf4bb994-nkvt7_30fee751-9e1e-483a-9e5d-d7eed8c53a49/manager/0.log"
Sep 11 00:12:32.099202 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:32.099164    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5hmzr_eefba04a-fca1-4281-9427-8635b54bd72a/payload-processing/0.log"
Sep 11 00:12:32.113776 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:32.113733    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c5d97497d-trwfb_12bf9c01-3b14-47e8-97a1-cd867914835f/router/0.log"
Sep 11 00:12:34.172718 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:34.172682    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-5hmzr_eefba04a-fca1-4281-9427-8635b54bd72a/payload-processing/0.log"
Sep 11 00:12:34.194190 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:34.194155    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c5d97497d-trwfb_12bf9c01-3b14-47e8-97a1-cd867914835f/router/0.log"
Sep 11 00:12:36.623526 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:36.623489    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-69fsw_96674fff-8032-41fb-b75e-ca10f7d60860/kube-storage-version-migrator-operator/1.log"
Sep 11 00:12:36.625480 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:36.625443    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-69fsw_96674fff-8032-41fb-b75e-ca10f7d60860/kube-storage-version-migrator-operator/0.log"
Sep 11 00:12:37.680798 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:37.680767    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2lmmn_2cf86d6f-6f67-40a4-98bf-f97c338816b3/kube-multus/0.log"
Sep 11 00:12:38.030355 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.030328    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll288_e48f204d-76c2-43fa-8100-486c6a3e31a3/kube-multus-additional-cni-plugins/0.log"
Sep 11 00:12:38.049626 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.049596    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll288_e48f204d-76c2-43fa-8100-486c6a3e31a3/egress-router-binary-copy/0.log"
Sep 11 00:12:38.068117 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.068093    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll288_e48f204d-76c2-43fa-8100-486c6a3e31a3/cni-plugins/0.log"
Sep 11 00:12:38.118040 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.118014    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll288_e48f204d-76c2-43fa-8100-486c6a3e31a3/bond-cni-plugin/0.log"
Sep 11 00:12:38.181322 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.181296    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll288_e48f204d-76c2-43fa-8100-486c6a3e31a3/routeoverride-cni/0.log"
Sep 11 00:12:38.229984 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.229954    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll288_e48f204d-76c2-43fa-8100-486c6a3e31a3/whereabouts-cni-bincopy/0.log"
Sep 11 00:12:38.251330 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.251310    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ll288_e48f204d-76c2-43fa-8100-486c6a3e31a3/whereabouts-cni/0.log"
Sep 11 00:12:38.335644 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.335583    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m8gj9_745744e6-2b42-4e82-bfba-6c2f79f57633/network-metrics-daemon/0.log"
Sep 11 00:12:38.349840 ip-10-0-137-218 kubenswrapper[2562]: I0911 00:12:38.349813    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m8gj9_745744e6-2b42-4e82-bfba-6c2f79f57633/kube-rbac-proxy/0.log"