Sep 10 17:21:52.792297 ip-10-0-142-242 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 17:21:52.792307 ip-10-0-142-242 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 17:21:52.792314 ip-10-0-142-242 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 17:21:52.792458 ip-10-0-142-242 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 17:22:02.898175 ip-10-0-142-242 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 17:22:02.898191 ip-10-0-142-242 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 13eed8dad4d1407bb09085d523ea1163 --
Sep 10 17:23:51.178504 ip-10-0-142-242 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 17:23:51.570313 ip-10-0-142-242 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:23:51.570313 ip-10-0-142-242 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 17:23:51.570313 ip-10-0-142-242 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:23:51.570313 ip-10-0-142-242 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 17:23:51.570313 ip-10-0-142-242 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:23:51.571365 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.571281    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 17:23:51.575785 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575771    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575787    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575791    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575794    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575796    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575800    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575803    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575806    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575809    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575812    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575814    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575817    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575819    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575822    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575825    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575828    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:51.575823 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575831    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575834    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575837    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575840    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575842    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575845    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575847    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575851    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575855    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575858    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575861    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575864    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575866    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575869    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575871    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575874    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575876    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575879    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575882    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575884    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:51.576218 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575887    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575889    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575899    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575901    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575905    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575909    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575912    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575915    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575918    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575921    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575924    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575927    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575929    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575932    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575935    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575938    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575941    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575943    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575946    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575948    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:51.576725 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575950    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575953    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575955    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575958    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575960    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575963    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575967    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575970    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575973    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575976    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575978    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575981    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575983    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575986    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575988    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575996    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.575999    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576001    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576004    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:51.577220 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576006    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576009    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576012    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576014    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576018    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576020    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576023    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576026    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576029    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576031    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576034    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576036    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576039    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576042    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576044    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576047    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576049    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576059    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576062    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576065    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:51.577701 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576068    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576070    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576073    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576076    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576899    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576906    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576910    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576912    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576915    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576925    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576928    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576931    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576933    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576936    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576939    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576943    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576945    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576948    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576950    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576954    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:51.578191 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576956    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576959    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576962    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576965    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576968    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576970    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576973    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576976    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576978    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576981    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576983    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576986    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576988    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576990    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576993    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576996    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.576998    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577000    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577003    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577006    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:51.578776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577008    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577011    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577019    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577021    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577024    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577026    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577030    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577032    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577035    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577037    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577040    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577045    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577049    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577052    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577054    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577057    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577060    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577063    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577065    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577067    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:51.579287 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577070    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577073    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577075    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577078    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577081    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577083    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577086    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577089    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577091    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577094    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577096    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577099    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577102    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577104    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577107    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577115    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577117    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577120    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577123    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577125    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:51.579787 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577128    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577130    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577133    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577136    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577139    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577142    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577145    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577147    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577150    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577152    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577155    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577157    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577160    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577162    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577165    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577168    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577170    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577172    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:51.580656 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577176    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577180    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577183    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577186    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577188    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577632    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577637    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577640    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577643    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577651    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577654    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577657    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577660    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577662    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577665    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577668    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577670    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577673    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577676    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577679    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:51.581423 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577681    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577684    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577686    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577688    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577691    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577693    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577696    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577698    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577701    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577704    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577706    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577709    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577712    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577714    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577717    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577720    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577722    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577724    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577727    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:51.581935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577729    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577731    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577734    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577742    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577744    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577747    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577751    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577754    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577756    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577759    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577762    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577766    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577770    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577773    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577775    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577778    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577781    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577783    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577786    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:51.582457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577789    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577792    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577795    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577797    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577799    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577802    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577805    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577807    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577810    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577828    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577833    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577836    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577839    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577842    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577845    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577849    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577852    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577857    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577860    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:51.582935 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577863    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577865    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577868    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577871    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577873    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577876    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577878    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577881    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577884    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577886    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577889    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577891    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577894    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577896    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577899    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577901    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577904    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577906    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577909    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577911    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:51.583419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577914    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577916    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577919    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577922    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577924    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577927    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.577930    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.577997    2568 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578028    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578037    2568 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578042    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578049    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578053    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578057    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578061    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578065    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578068    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578071    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578075    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578078    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578081    2568 flags.go:64] FLAG: --cgroup-root=""
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578084    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578087    2568 flags.go:64] FLAG: --client-ca-file=""
Sep 10 17:23:51.583952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578090    2568 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578092    2568 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578097    2568 flags.go:64] FLAG: --cluster-domain=""
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578103    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578106    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578109    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578112    2568 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578116    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578118    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578126    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578129    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578132    2568 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578135    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578138    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578141    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578144    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578148    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578151    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578154    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578157    2568 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578160    2568 flags.go:64] FLAG: --enable-server="true"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578164    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578852    2568 flags.go:64] FLAG: --event-burst="100"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578856    2568 flags.go:64] FLAG: --event-qps="50"
Sep 10 17:23:51.584550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578859    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578863    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578866    2568 flags.go:64] FLAG: --eviction-hard=""
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578870    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578873    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578876    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578879    2568 flags.go:64] FLAG: --eviction-soft=""
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578882    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578886    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578889    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578892    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578895    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578897    2568 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578900    2568 flags.go:64] FLAG: --feature-gates=""
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578904    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578907    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578910    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578915    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578918    2568 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578921    2568 flags.go:64] FLAG: --help="false"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578924    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578927    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578930    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 17:23:51.585131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578933    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578936    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578939    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578942    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578945    2568 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578947    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578950    2568 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578954    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578957    2568 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578960    2568 flags.go:64] FLAG: --kube-reserved=""
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578962    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578965    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578968    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578971    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578974    2568 flags.go:64] FLAG: --lock-file=""
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578976    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578979    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578986    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578992    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578995    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.578997    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579000    2568 flags.go:64] FLAG: --logging-format="text"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579003    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579007    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 17:23:51.585742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579009    2568 flags.go:64] FLAG: --manifest-url=""
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579012    2568 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579018    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579021    2568 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579026    2568 flags.go:64] FLAG: --max-pods="110"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579029    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579031    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579034    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579037    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579040    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579043    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579046    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579053    2568 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579056    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579058    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579061    2568 flags.go:64] FLAG: --pod-cidr=""
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579064    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579070    2568 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579073    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579076    2568 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579079    2568 flags.go:64] FLAG: --port="10250"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579082    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579085    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06d73d9d81b8d1c8d"
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579088    2568 flags.go:64] FLAG: --qos-reserved=""
Sep 10 17:23:51.586336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579091    2568 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579095    2568 flags.go:64] FLAG: --register-node="true"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579098    2568 flags.go:64] FLAG: --register-with-taints=""
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579101    2568 flags.go:64] FLAG: --registry-burst="10"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579104    2568 flags.go:64] FLAG: --registry-qps="5"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579107    2568 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579110    2568 flags.go:64] FLAG: --reserved-memory=""
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579113    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579116    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579120    2568 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579123    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579126    2568 flags.go:64] FLAG: --runonce="false"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579129    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579132    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579135    2568 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579138    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579141    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579144    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579147    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579150    2568 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579153    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579156    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579158    2568 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579161    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579164    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 17:23:51.586924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579177    2568 flags.go:64] FLAG: --system-cgroups=""
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579180    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579185    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579188    2568 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579191    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579199    2568 flags.go:64] FLAG: --tls-min-version=""
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579202    2568 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579205    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579209    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579212    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579215    2568 flags.go:64] FLAG: --v="2"
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579219    2568 flags.go:64] FLAG: --version="false"
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579223    2568 flags.go:64] FLAG: --vmodule=""
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579226    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.579229    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579419    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579423    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579427    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579431    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579434    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579437    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579440    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579443    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:51.587584 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579446    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579448    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579451    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579453    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579456    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579459    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579461    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579464    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579468    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579471    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579480    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579483    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579486    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579488    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579492    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579494    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579497    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579501    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579504    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:51.588229 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579507    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579510    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579513    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579516    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579518    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579522    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579524    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579527    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579531    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579535    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579538    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579541    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579544    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579547    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579549    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579552    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579555    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579558    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579560    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:51.588718 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579563    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579566    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579569    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579571    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579574    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579583    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579586    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579588    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579591    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579593    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579596    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579599    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579602    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579606    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579609    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579612    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579615    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579618    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579620    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579623    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:51.589196 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579625    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579629    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579632    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579634    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579637    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579640    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579642    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579645    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579648    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579650    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579653    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579655    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579658    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579660    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579663    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579665    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579668    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579670    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.579719    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581285    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:51.589771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581298    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581304    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581310    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581318    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581389    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581409    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581416    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581422    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581427    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581433    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581438    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581443    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.581448    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.582095    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.589794    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 17:23:51.590280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.589812    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.590565    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.590572    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590635    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590640    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590643    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590647    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590651    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590654    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590657    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590661    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590666    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590669    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590671    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590675    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:51.590671 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590678    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590681    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590684    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590687    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590689    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590692    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590695    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590700    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590703    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590706    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590709    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590711    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590714    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590716    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590719    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590722    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590725    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590727    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590730    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590732    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:51.591039 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590735    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590737    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590740    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590743    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590745    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590748    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590751    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590753    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590756    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590759    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590761    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590764    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590767    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590770    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590773    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590776    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590778    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590781    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590784    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590786    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:51.591562 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590789    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590791    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590794    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590796    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590799    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590801    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590804    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590807    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590809    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590812    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590814    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590817    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590820    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590822    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590825    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590828    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590830    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590833    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590835    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590838    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:51.592072 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590841    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590844    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590847    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590849    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590852    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590855    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590858    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590861    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590863    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590866    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590870    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590873    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590876    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590879    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590883    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590886    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590890    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590893    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590896    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590899    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590902    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590904    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590907    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590910    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590913    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590916    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.590919    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.590924    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591044    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591050    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591056    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591059    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591062    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:23:51.593069 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591065    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591068    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591070    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591073    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591076    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591079    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591081    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591084    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591087    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591089    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591092    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591095    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591097    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591100    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591102    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591105    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591108    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591110    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591113    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591116    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:23:51.593440 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591118    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591121    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591123    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591126    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591128    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591131    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591134    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591136    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591139    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591141    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591144    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591147    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591150    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591153    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591155    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591158    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591161    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591163    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591166    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591169    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:23:51.593942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591172    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591174    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591177    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591179    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591182    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591184    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591187    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591189    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591192    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591194    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591197    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591199    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591202    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591204    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591207    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591209    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591212    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591215    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591217    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591220    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:23:51.594454 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591222    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591224    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591227    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591230    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591246    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591249    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591252    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591255    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591258    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591260    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591263    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591265    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591268    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591271    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591273    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591276    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591280    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591283    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591286    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:23:51.594945 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591288    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591291    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591294    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591297    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591300    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591303    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591306    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591308    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591312    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591315    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591318    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591321    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591324    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591326    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:23:51.595447 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:51.591329    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:23:51.595809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.591334    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:23:51.595809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.591476    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 17:23:51.595809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.595066    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 17:23:51.596543 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.596531    2568 server.go:1019] "Starting client certificate rotation"
Sep 10 17:23:51.596649 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.596630    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:23:51.596689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.596664    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:23:51.599659 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.599645    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:23:51.600818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.600790    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:23:51.613739 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.613714    2568 log.go:25] "Validated CRI v1 runtime API"
Sep 10 17:23:51.618496 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.618475    2568 log.go:25] "Validated CRI v1 image API"
Sep 10 17:23:51.619455 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.619438    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 17:23:51.626134 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.626111    2568 fs.go:135] Filesystem UUIDs: map[30abb0fc-3da1-41ba-83be-e3fc5055a67b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bcf75d63-11e6-490d-8f77-66ff9906cc5c:/dev/nvme0n1p4]
Sep 10 17:23:51.626208 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.626133    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 17:23:51.629401 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.629384    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:23:51.631658 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.631541    2568 manager.go:216] Machine: {Timestamp:2026-09-10 17:23:51.63015553 +0000 UTC m=+0.339749788 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108672 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f00cf054adf64344c10f12449deed SystemUUID:ec2f00cf-054a-df64-344c-10f12449deed BootID:13eed8da-d4d1-407b-b090-85d523ea1163 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:9a:da:49:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:9a:da:49:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:7e:eb:17:07:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 17:23:51.631658 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.631653    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 17:23:51.631802 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.631740    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 17:23:51.635086 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.635060    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 17:23:51.635270 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.635089    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 17:23:51.636021 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.636009    2568 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 17:23:51.636087 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.636029    2568 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 17:23:51.636087 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.636048    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:23:51.636830 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.636818    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:23:51.638246 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.638221    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 17:23:51.641073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641059    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:23:51.641216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641205    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 17:23:51.641286 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641261    2568 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 17:23:51.641347 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641315    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 17:23:51.641347 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641335    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 17:23:51.641433 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641349    2568 kubelet.go:398] "Adding apiserver pod source"
Sep 10 17:23:51.641433 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641361    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 17:23:51.641765 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641751    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:23:51.641824 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.641775    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:23:51.644194 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.644177    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 17:23:51.646465 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.646449    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 17:23:51.646524 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.646487    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 17:23:51.647228 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647217    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 17:23:51.647284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647247    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 17:23:51.647284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647255    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 17:23:51.647284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647272    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 17:23:51.647284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647284    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 17:23:51.647409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647300    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 17:23:51.647409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647313    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 17:23:51.647409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647319    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 17:23:51.647409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647325    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 17:23:51.647409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647331    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 17:23:51.647409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647343    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 17:23:51.647409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.647348    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 17:23:51.648846 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.648832    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 17:23:51.648882 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.648848    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 17:23:51.650637 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.650593    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 17:23:51.650699 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.650628    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 17:23:51.652809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.652796    2568 server.go:1268] "Started kubelet"
Sep 10 17:23:51.652913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.652873    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 17:23:51.653001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.652897    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 17:23:51.653001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.652979    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 17:23:51.653664 ip-10-0-142-242 systemd[1]: Started Kubernetes Kubelet.
Sep 10 17:23:51.655073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.654794    2568 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 17:23:51.656486 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.656470    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 17:23:51.661185 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.661164    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 17:23:51.661339 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.661320    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 17:23:51.661522 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.661503    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:51.661662 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.661641    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 17:23:51.661845 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.661795    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 17:23:51.662013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.661995    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 17:23:51.662013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.662000    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 17:23:51.662013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.662017    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 17:23:51.662193 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.662159    2568 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 17:23:51.662193 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.662173    2568 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 17:23:51.662637 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.661728    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-242.ec2.internal.18d4058789282b00  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-242.ec2.internal,UID:ip-10-0-142-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-242.ec2.internal,},FirstTimestamp:2026-09-10 17:23:51.652764416 +0000 UTC m=+0.362358673,LastTimestamp:2026-09-10 17:23:51.652764416 +0000 UTC m=+0.362358673,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-242.ec2.internal,}"
Sep 10 17:23:51.662722 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.662703    2568 factory.go:55] Registering systemd factory
Sep 10 17:23:51.662784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.662734    2568 factory.go:223] Registration of the systemd container factory successfully
Sep 10 17:23:51.662961 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.662948    2568 factory.go:153] Registering CRI-O factory
Sep 10 17:23:51.662994 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.662964    2568 factory.go:223] Registration of the crio container factory successfully
Sep 10 17:23:51.663024 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.663011    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 17:23:51.663059 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.663026    2568 factory.go:103] Registering Raw factory
Sep 10 17:23:51.663059 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.663038    2568 manager.go:1196] Started watching for new ooms in manager
Sep 10 17:23:51.663380 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.663363    2568 manager.go:349] Starting recovery of all containers
Sep 10 17:23:51.667289 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.667268    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 17:23:51.667372 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.667329    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 17:23:51.673439 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.673416    2568 manager.go:354] Recovery completed
Sep 10 17:23:51.677966 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.677852    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:51.679545 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.679509    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kq6p7"
Sep 10 17:23:51.681249 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.681215    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:51.681341 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.681269    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:51.681341 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.681299    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:51.681960 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.681945    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 17:23:51.681960 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.681960    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 17:23:51.682081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.681979    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:23:51.682081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.681978    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:51.682816 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.682799    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:51.682816 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.682834    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:51.682923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.682849    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:51.683080 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.683067    2568 policy_none.go:49] "None policy: Start"
Sep 10 17:23:51.683132 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.683086    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 17:23:51.683132 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.683100    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 17:23:51.683602 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.683536    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-242.ec2.internal.18d405878adae4c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-242.ec2.internal,UID:ip-10-0-142-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-242.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-242.ec2.internal,},FirstTimestamp:2026-09-10 17:23:51.681254596 +0000 UTC m=+0.390848855,LastTimestamp:2026-09-10 17:23:51.681254596 +0000 UTC m=+0.390848855,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-242.ec2.internal,}"
Sep 10 17:23:51.684265 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.684252    2568 policy_none.go:47] "Start"
Sep 10 17:23:51.689461 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.689436    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kq6p7"
Sep 10 17:23:51.691261 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.691190    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-242.ec2.internal.18d405878adb3506  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-242.ec2.internal,UID:ip-10-0-142-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-242.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-242.ec2.internal,},FirstTimestamp:2026-09-10 17:23:51.681275142 +0000 UTC m=+0.390869399,LastTimestamp:2026-09-10 17:23:51.681275142 +0000 UTC m=+0.390869399,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-242.ec2.internal,}"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.712722    2568 manager.go:337] "Starting Device Plugin manager"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.712756    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.712770    2568 server.go:87] "Starting device plugin registration server"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.713020    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.713033    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.713191    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.713280    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.713288    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.713703    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 17:23:51.722515 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.713757    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:51.755570 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.755537    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 17:23:51.756752 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.756735    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 17:23:51.756861 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.756782    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 17:23:51.756861 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.756807    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 17:23:51.756861 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.756853    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 17:23:51.759207 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.759181    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:51.813282 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.813250    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:51.814378 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.814359    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:51.814447 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.814392    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:51.814447 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.814403    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:51.814447 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.814425    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.820989 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.820946    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.820989 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.820968    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-242.ec2.internal\": node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:51.837207 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.837180    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:51.857872 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.857840    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"]
Sep 10 17:23:51.857960 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.857924    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:51.859339 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.859323    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:51.859413 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.859350    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:51.859413 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.859360    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:51.860613 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.860602    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:51.860734 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.860721    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.860772 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.860749    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:51.861541 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.861521    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:51.861541 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.861542    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:51.861647 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.861551    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:51.861647 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.861521    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:51.861647 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.861622    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:51.861647 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.861634    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:51.862812 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.862797    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.862903 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.862822    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:23:51.863189 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.863173    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7091a140b9b7cb44dee6eef96e11a133-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"7091a140b9b7cb44dee6eef96e11a133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.863273 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.863203    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7091a140b9b7cb44dee6eef96e11a133-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"7091a140b9b7cb44dee6eef96e11a133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.863273 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.863227    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f3333ec3c604b826ba36ee4849cfacb-config\") pod \"kube-apiserver-proxy-ip-10-0-142-242.ec2.internal\" (UID: \"1f3333ec3c604b826ba36ee4849cfacb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.863536 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.863513    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:23:51.863585 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.863539    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:23:51.863585 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.863554    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:23:51.880795 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.880773    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-242.ec2.internal\" not found" node="ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.884908 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.884893    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-242.ec2.internal\" not found" node="ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.938288 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:51.938265    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:51.964201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.964178    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7091a140b9b7cb44dee6eef96e11a133-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"7091a140b9b7cb44dee6eef96e11a133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.964293 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.964203    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7091a140b9b7cb44dee6eef96e11a133-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"7091a140b9b7cb44dee6eef96e11a133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.964293 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.964219    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f3333ec3c604b826ba36ee4849cfacb-config\") pod \"kube-apiserver-proxy-ip-10-0-142-242.ec2.internal\" (UID: \"1f3333ec3c604b826ba36ee4849cfacb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.964293 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.964290    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f3333ec3c604b826ba36ee4849cfacb-config\") pod \"kube-apiserver-proxy-ip-10-0-142-242.ec2.internal\" (UID: \"1f3333ec3c604b826ba36ee4849cfacb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.964427 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.964316    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7091a140b9b7cb44dee6eef96e11a133-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"7091a140b9b7cb44dee6eef96e11a133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:51.964427 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:51.964353    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7091a140b9b7cb44dee6eef96e11a133-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal\" (UID: \"7091a140b9b7cb44dee6eef96e11a133\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:52.039165 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.039132    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:52.139804 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.139732    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:52.183416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.183382    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:52.188159 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.188030    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:52.240639 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.240609    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:52.341074 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.341043    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:52.441564 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.441491    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:52.541607 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.541570    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-242.ec2.internal\" not found"
Sep 10 17:23:52.545066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.545048    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:52.562375 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.562349    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:52.573692 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.573666    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:23:52.575510 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.575491    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal"
Sep 10 17:23:52.587756 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.587736    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:23:52.596330 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.596301    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 17:23:52.596479 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.596459    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:23:52.596534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.596497    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:23:52.641905 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.641876    2568 apiserver.go:52] "Watching apiserver"
Sep 10 17:23:52.661938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.661911    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 17:23:52.668187 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.668161    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 17:23:52.668605 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.668579    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-pd8gz","kube-system/konnectivity-agent-pntqj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57","openshift-multus/network-metrics-daemon-gl4v7","openshift-network-diagnostics/network-check-target-njlj5","openshift-network-operator/iptables-alerter-kglhk","openshift-ovn-kubernetes/ovnkube-node-znlbh","kube-system/global-pull-secret-syncer-5gjsv","kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5r7kc","openshift-dns/node-resolver-2pdpk","openshift-image-registry/node-ca-2rtwg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal","openshift-multus/multus-additional-cni-plugins-2r8ch"]
Sep 10 17:23:52.671951 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.671927    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.674115 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.674094    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.676710 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.676688    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.678707 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.678684    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 17:23:52.678707 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.678699    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nqlkp\""
Sep 10 17:23:52.678855 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.678688    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 17:23:52.678855 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.678794    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 17:23:52.678855 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.678689    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 17:23:52.678855 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.678692    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 17:23:52.679061 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.678938    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 17:23:52.679061 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.678969    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 17:23:52.679164 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.679127    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 17:23:52.679220 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.679181    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:52.679270 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.679217    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-spzjl\""
Sep 10 17:23:52.679400 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.679316    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:23:52.680137 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.680121    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 17:23:52.680301 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.680286    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 17:23:52.680383 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.680288    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 17:23:52.680383 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.680349    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-brdfz\""
Sep 10 17:23:52.681660 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.681645    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:52.681727 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.681711    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:23:52.682714 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.682694    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:23:52.684002 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.683984    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.686438 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.686413    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.687404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.687385    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 17:23:52.687523 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.687505    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:23:52.687590 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.687572    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qgqgm\""
Sep 10 17:23:52.687645 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.687633    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 17:23:52.688856 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.688834    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 17:23:52.689448 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.689433    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ktq8c\""
Sep 10 17:23:52.689551 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.689538    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.689607 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.689592    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:23:52.689778 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.689761    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 17:23:52.689828 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.689779    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 17:23:52.689924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.689911    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 17:23:52.690767 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.690753    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 17:23:52.690833 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.690795    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 17:18:52 +0000 UTC" deadline="2028-03-11 05:16:09.134400388 +0000 UTC"
Sep 10 17:23:52.690833 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.690820    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13139h52m16.443586206s"
Sep 10 17:23:52.691877 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.691845    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:23:52.694867 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.694420    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.696132 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.696110    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 17:23:52.696258 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.696139    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 17:23:52.696381 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.696364    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ttrt7\""
Sep 10 17:23:52.698496 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.698474    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.699618 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.699598    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:23:52.699751 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.699732    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 17:23:52.700863 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.700845    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.701983 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.701966    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7r266\""
Sep 10 17:23:52.702462 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.702429    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 17:23:52.702564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.702520    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-749sc\""
Sep 10 17:23:52.703804 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.703789    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 17:23:52.705793 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.705773    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sjxpm\""
Sep 10 17:23:52.705793 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.705775    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 17:23:52.705902 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.705776    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 17:23:52.730060 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.730040    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4qzgv"
Sep 10 17:23:52.744977 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.744960    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4qzgv"
Sep 10 17:23:52.745265 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.745247    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:52.763026 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.763009    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 17:23:52.766310 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:52.766277    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7091a140b9b7cb44dee6eef96e11a133.slice/crio-d688ae64e67ca9ad81fe49c7f3c4e50bfc85fec41b4f3d75a3c7d5d2b81f38b7 WatchSource:0}: Error finding container d688ae64e67ca9ad81fe49c7f3c4e50bfc85fec41b4f3d75a3c7d5d2b81f38b7: Status 404 returned error can't find the container with id d688ae64e67ca9ad81fe49c7f3c4e50bfc85fec41b4f3d75a3c7d5d2b81f38b7
Sep 10 17:23:52.766551 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:52.766529    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3333ec3c604b826ba36ee4849cfacb.slice/crio-d48a92e5f867c899bb93dadd64e76f1b4e56527bc6e57752033b46e267e6eee6 WatchSource:0}: Error finding container d48a92e5f867c899bb93dadd64e76f1b4e56527bc6e57752033b46e267e6eee6: Status 404 returned error can't find the container with id d48a92e5f867c899bb93dadd64e76f1b4e56527bc6e57752033b46e267e6eee6
Sep 10 17:23:52.768392 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768366    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-cni-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.768493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768405    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-etc-kubernetes\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.768493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768430    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-ovn\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.768493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768471    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.768627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768500    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-env-overrides\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.768627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768523    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-hosts-file\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.768627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768544    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysconfig\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.768627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-lib-modules\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768642    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78220b5d-9112-416d-a65f-c2c7f3359167-tmp\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768670    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-netns\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768696    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768723    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-system-cni-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768747    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-cnibin\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768768    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grh2r\" (UniqueName: \"kubernetes.io/projected/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-kube-api-access-grh2r\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768783    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-run-netns\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768800    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovnkube-script-lib\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.768818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768818    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-system-cni-dir\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768840    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-modprobe-d\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768854    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-cni-bin\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-serviceca\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768893    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ced14a44-5fd5-4179-af31-eef5392a3e7c-kubelet-config\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768924    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ced14a44-5fd5-4179-af31-eef5392a3e7c-dbus\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768951    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-systemd-units\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.768968    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-systemd\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769002    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-cni-multus\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769025    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnff\" (UniqueName: \"kubernetes.io/projected/39d4f881-f1f5-48ac-ae30-401b708297d2-kube-api-access-9pnff\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769041    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571-agent-certs\") pod \"konnectivity-agent-pntqj\" (UID: \"8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571\") " pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769056    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-kubelet\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769082    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769125    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-tmp-dir\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.769201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769165    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769221    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-k8s-cni-cncf-io\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769283    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-device-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769309    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkspz\" (UniqueName: \"kubernetes.io/projected/d080ad12-87b4-483f-b0b8-5ab978b22216-kube-api-access-xkspz\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769350    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769375    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-cni-netd\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769392    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxpk\" (UniqueName: \"kubernetes.io/projected/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-kube-api-access-lzxpk\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769407    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysctl-conf\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769427    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-systemd\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769445    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9038897d-9736-4d81-a7b8-ea5b7d52e587-cni-binary-copy\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769461    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-conf-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769475    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-host\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769490    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769512    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-node-log\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769527    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-cni-bin\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769544    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-var-lib-kubelet\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.769781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769558    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-kubelet\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769694    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-registration-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769727    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-sys-fs\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769752    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769794    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cni-binary-copy\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769817    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-sys\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769840    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-run\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769873    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-host\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769919    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.769958    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-etc-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770004    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cnibin\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770031    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs47z\" (UniqueName: \"kubernetes.io/projected/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-kube-api-access-hs47z\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770076    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvfp\" (UniqueName: \"kubernetes.io/projected/78220b5d-9112-416d-a65f-c2c7f3359167-kube-api-access-djvfp\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770119    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-daemon-config\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770144    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-var-lib-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770169    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-log-socket\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.770623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770192    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78220b5d-9112-416d-a65f-c2c7f3359167-etc-tuned\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770215    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-socket-dir-parent\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770238    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-socket-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770281    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpq2n\" (UniqueName: \"kubernetes.io/projected/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-kube-api-access-dpq2n\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770305    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovn-node-metrics-cert\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770328    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-kubernetes\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770350    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysctl-d\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770376    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-multus-certs\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770399    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlbj\" (UniqueName: \"kubernetes.io/projected/9038897d-9736-4d81-a7b8-ea5b7d52e587-kube-api-access-cwlbj\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770422    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d080ad12-87b4-483f-b0b8-5ab978b22216-iptables-alerter-script\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770448    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d080ad12-87b4-483f-b0b8-5ab978b22216-host-slash\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-hostroot\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770531    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571-konnectivity-ca\") pod \"konnectivity-agent-pntqj\" (UID: \"8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571\") " pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770557    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-slash\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770582    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-run-ovn-kubernetes\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770621    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovnkube-config\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.771719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770646    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.772332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770671    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.772332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.770696    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r4td\" (UniqueName: \"kubernetes.io/projected/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-kube-api-access-7r4td\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.772332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.771249    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:23:52.870997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.870961    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78220b5d-9112-416d-a65f-c2c7f3359167-etc-tuned\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.870997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.870997    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-socket-dir-parent\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871014    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-socket-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871029    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpq2n\" (UniqueName: \"kubernetes.io/projected/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-kube-api-access-dpq2n\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871045    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovn-node-metrics-cert\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871062    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-kubernetes\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysctl-d\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871108    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-multus-certs\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871131    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlbj\" (UniqueName: \"kubernetes.io/projected/9038897d-9736-4d81-a7b8-ea5b7d52e587-kube-api-access-cwlbj\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871154    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d080ad12-87b4-483f-b0b8-5ab978b22216-iptables-alerter-script\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871177    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d080ad12-87b4-483f-b0b8-5ab978b22216-host-slash\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.871209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871201    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-multus-certs\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871221    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-hostroot\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871201    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-socket-dir-parent\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871235    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-kubernetes\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871278    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-hostroot\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871310    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysctl-d\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871361    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d080ad12-87b4-483f-b0b8-5ab978b22216-host-slash\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871379    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571-konnectivity-ca\") pod \"konnectivity-agent-pntqj\" (UID: \"8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571\") " pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871436    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-slash\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871473    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-run-ovn-kubernetes\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871499    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-slash\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871506    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovnkube-config\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871562    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871591    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-run-ovn-kubernetes\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871596    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871647    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r4td\" (UniqueName: \"kubernetes.io/projected/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-kube-api-access-7r4td\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-cni-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.871719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871679    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871697    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-etc-kubernetes\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871723    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-ovn\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871750    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871778    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-env-overrides\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871803    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-hosts-file\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871827    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysconfig\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871852    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-lib-modules\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871874    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78220b5d-9112-416d-a65f-c2c7f3359167-tmp\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871750    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-socket-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871899    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-netns\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871928    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871954    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-system-cni-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871976    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-cnibin\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872003    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grh2r\" (UniqueName: \"kubernetes.io/projected/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-kube-api-access-grh2r\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872024    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovnkube-config\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872031    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-run-netns\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872048    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-cni-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.872564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872056    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571-konnectivity-ca\") pod \"konnectivity-agent-pntqj\" (UID: \"8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571\") " pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872062    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-netns\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872023    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-etc-kubernetes\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872059    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovnkube-script-lib\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.871976    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d080ad12-87b4-483f-b0b8-5ab978b22216-iptables-alerter-script\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872135    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysconfig\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872145    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-cnibin\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872134    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872171    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872205    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-lib-modules\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872217    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-hosts-file\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872258    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-run-netns\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-ovn\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872339    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-system-cni-dir\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872349    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-system-cni-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872371    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-modprobe-d\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872399    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-cni-bin\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872404    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-system-cni-dir\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.873446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-serviceca\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872451    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ced14a44-5fd5-4179-af31-eef5392a3e7c-kubelet-config\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872461    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-cni-bin\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872453    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872487    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-modprobe-d\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872478    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ced14a44-5fd5-4179-af31-eef5392a3e7c-dbus\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872545    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ced14a44-5fd5-4179-af31-eef5392a3e7c-dbus\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872551    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-env-overrides\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872554    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-systemd-units\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872598    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-systemd\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872630    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-cni-multus\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872658    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnff\" (UniqueName: \"kubernetes.io/projected/39d4f881-f1f5-48ac-ae30-401b708297d2-kube-api-access-9pnff\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872658    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872699    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-systemd\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872721    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-cni-multus\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872734    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovnkube-script-lib\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872775    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-systemd-units\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872735    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571-agent-certs\") pod \"konnectivity-agent-pntqj\" (UID: \"8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571\") " pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:23:52.874215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872825    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ced14a44-5fd5-4179-af31-eef5392a3e7c-kubelet-config\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872847    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-kubelet\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872877    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872896    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-serviceca\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872903    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-tmp-dir\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872927    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872947    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-kubelet\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872953    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-k8s-cni-cncf-io\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872966    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-run-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.872979    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-device-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873007    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkspz\" (UniqueName: \"kubernetes.io/projected/d080ad12-87b4-483f-b0b8-5ab978b22216-kube-api-access-xkspz\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873013    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-run-k8s-cni-cncf-io\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873059    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-cni-netd\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873064    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-device-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873070    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxpk\" (UniqueName: \"kubernetes.io/projected/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-kube-api-access-lzxpk\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.875216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873111    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysctl-conf\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873116    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-cni-netd\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.873151    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-systemd\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873234    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-sysctl-conf\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873244    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-etc-systemd\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.873267    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret podName:ced14a44-5fd5-4179-af31-eef5392a3e7c nodeName:}" failed. No retries permitted until 2026-09-10 17:23:53.373206209 +0000 UTC m=+2.082800477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret") pod "global-pull-secret-syncer-5gjsv" (UID: "ced14a44-5fd5-4179-af31-eef5392a3e7c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873352    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9038897d-9736-4d81-a7b8-ea5b7d52e587-cni-binary-copy\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873385    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-conf-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873407    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-host\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873428    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-conf-dir\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873431    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873469    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-host\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873479    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-node-log\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873503    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-cni-bin\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873512    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-node-log\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873525    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-var-lib-kubelet\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873481    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873572    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-kubelet\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873588    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-host-cni-bin\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873597    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-registration-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873625    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-sys-fs\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873633    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-var-lib-kubelet\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873662    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873676    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-registration-dir\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873684    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cni-binary-copy\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873706    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-sys\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873711    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9038897d-9736-4d81-a7b8-ea5b7d52e587-host-var-lib-kubelet\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873726    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-run\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873749    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-host\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873771    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873792    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-etc-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873810    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39d4f881-f1f5-48ac-ae30-401b708297d2-sys-fs\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873817    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cnibin\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.876782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873852    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs47z\" (UniqueName: \"kubernetes.io/projected/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-kube-api-access-hs47z\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873853    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-host\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.873988    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9038897d-9736-4d81-a7b8-ea5b7d52e587-cni-binary-copy\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874053    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-etc-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.873774    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.874129    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs podName:3ea78e61-4de5-43fa-81a2-8ffff13e1d3a nodeName:}" failed. No retries permitted until 2026-09-10 17:23:53.374111391 +0000 UTC m=+2.083705661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs") pod "network-metrics-daemon-gl4v7" (UID: "3ea78e61-4de5-43fa-81a2-8ffff13e1d3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874067    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cnibin\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874164    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-tmp-dir\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874192    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-sys\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874234    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvfp\" (UniqueName: \"kubernetes.io/projected/78220b5d-9112-416d-a65f-c2c7f3359167-kube-api-access-djvfp\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874244    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78220b5d-9112-416d-a65f-c2c7f3359167-run\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-daemon-config\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874288    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-var-lib-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874331    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-log-socket\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874374    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-cni-binary-copy\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874394    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-log-socket\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874435    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-var-lib-openvswitch\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.877297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.874822    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9038897d-9736-4d81-a7b8-ea5b7d52e587-multus-daemon-config\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.877808 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.876412    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78220b5d-9112-416d-a65f-c2c7f3359167-etc-tuned\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.877808 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.876447    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78220b5d-9112-416d-a65f-c2c7f3359167-tmp\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.877808 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.876610    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-ovn-node-metrics-cert\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.877808 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.876652    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571-agent-certs\") pod \"konnectivity-agent-pntqj\" (UID: \"8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571\") " pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:23:52.907481 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.907448    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpq2n\" (UniqueName: \"kubernetes.io/projected/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-kube-api-access-dpq2n\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:52.911223 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.911195    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r4td\" (UniqueName: \"kubernetes.io/projected/f7bc1e14-5460-41d2-a808-8ebb92f3c29f-kube-api-access-7r4td\") pod \"ovnkube-node-znlbh\" (UID: \"f7bc1e14-5460-41d2-a808-8ebb92f3c29f\") " pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:52.911926 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.911902    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxpk\" (UniqueName: \"kubernetes.io/projected/f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54-kube-api-access-lzxpk\") pod \"node-resolver-2pdpk\" (UID: \"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54\") " pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:52.912058 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.912037    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlbj\" (UniqueName: \"kubernetes.io/projected/9038897d-9736-4d81-a7b8-ea5b7d52e587-kube-api-access-cwlbj\") pod \"multus-pd8gz\" (UID: \"9038897d-9736-4d81-a7b8-ea5b7d52e587\") " pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:52.914151 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.914129    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:52.914241 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.914152    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:52.914241 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.914168    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lh8nc for pod openshift-network-diagnostics/network-check-target-njlj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:52.914404 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:52.914251    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc podName:cbade337-a7dc-4cab-9479-2303873983b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:53.414231638 +0000 UTC m=+2.123825901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8nc" (UniqueName: "kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc") pod "network-check-target-njlj5" (UID: "cbade337-a7dc-4cab-9479-2303873983b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:52.914681 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.914657    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnff\" (UniqueName: \"kubernetes.io/projected/39d4f881-f1f5-48ac-ae30-401b708297d2-kube-api-access-9pnff\") pod \"aws-ebs-csi-driver-node-6xl57\" (UID: \"39d4f881-f1f5-48ac-ae30-401b708297d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:52.915325 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.915305    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvfp\" (UniqueName: \"kubernetes.io/projected/78220b5d-9112-416d-a65f-c2c7f3359167-kube-api-access-djvfp\") pod \"tuned-5r7kc\" (UID: \"78220b5d-9112-416d-a65f-c2c7f3359167\") " pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:52.916053 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.916027    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs47z\" (UniqueName: \"kubernetes.io/projected/5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7-kube-api-access-hs47z\") pod \"multus-additional-cni-plugins-2r8ch\" (UID: \"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7\") " pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:52.916298 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.916279    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkspz\" (UniqueName: \"kubernetes.io/projected/d080ad12-87b4-483f-b0b8-5ab978b22216-kube-api-access-xkspz\") pod \"iptables-alerter-kglhk\" (UID: \"d080ad12-87b4-483f-b0b8-5ab978b22216\") " pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:52.916444 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.916426    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grh2r\" (UniqueName: \"kubernetes.io/projected/63e68c07-a8f9-4182-a40c-a1ef1470e7cc-kube-api-access-grh2r\") pod \"node-ca-2rtwg\" (UID: \"63e68c07-a8f9-4182-a40c-a1ef1470e7cc\") " pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:52.978993 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.978927    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:52.999626 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:52.999603    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:23:53.009397 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.009377    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2pdpk"
Sep 10 17:23:53.010634 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.010607    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bc1e14_5460_41d2_a808_8ebb92f3c29f.slice/crio-7cced750df23224789083193f8453dfdaf44b1cb13faf30f9b8010d68d5eafbe WatchSource:0}: Error finding container 7cced750df23224789083193f8453dfdaf44b1cb13faf30f9b8010d68d5eafbe: Status 404 returned error can't find the container with id 7cced750df23224789083193f8453dfdaf44b1cb13faf30f9b8010d68d5eafbe
Sep 10 17:23:53.018728 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.018704    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39fb70c_4b7c_4983_a5a1_9ed4d9fa1d54.slice/crio-7ea674e42035b202b8254921868108b629ee5043ce7daf6f4ee5eec94effbdb2 WatchSource:0}: Error finding container 7ea674e42035b202b8254921868108b629ee5043ce7daf6f4ee5eec94effbdb2: Status 404 returned error can't find the container with id 7ea674e42035b202b8254921868108b629ee5043ce7daf6f4ee5eec94effbdb2
Sep 10 17:23:53.034629 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.034599    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57"
Sep 10 17:23:53.040629 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.040610    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kglhk"
Sep 10 17:23:53.042691 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.042671    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d4f881_f1f5_48ac_ae30_401b708297d2.slice/crio-b55bf2324aa9c4226ba58741d2089c0d084648b814314edd08ca235e6a2a8caa WatchSource:0}: Error finding container b55bf2324aa9c4226ba58741d2089c0d084648b814314edd08ca235e6a2a8caa: Status 404 returned error can't find the container with id b55bf2324aa9c4226ba58741d2089c0d084648b814314edd08ca235e6a2a8caa
Sep 10 17:23:53.048713 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.048690    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd080ad12_87b4_483f_b0b8_5ab978b22216.slice/crio-e03272a77ae82929a6451cb2eb688bcf8471fbdaef5f1141a82ac92544c40a9e WatchSource:0}: Error finding container e03272a77ae82929a6451cb2eb688bcf8471fbdaef5f1141a82ac92544c40a9e: Status 404 returned error can't find the container with id e03272a77ae82929a6451cb2eb688bcf8471fbdaef5f1141a82ac92544c40a9e
Sep 10 17:23:53.066820 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.066798    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2r8ch"
Sep 10 17:23:53.073951 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.073933    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:23:53.075378 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.075355    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9997a7_c41e_4004_9f12_a6bc9bfe4ec7.slice/crio-17bebb4940e3b13246a412b8633bf8e5a4dc13f58d59cc4a3c52901231a484a1 WatchSource:0}: Error finding container 17bebb4940e3b13246a412b8633bf8e5a4dc13f58d59cc4a3c52901231a484a1: Status 404 returned error can't find the container with id 17bebb4940e3b13246a412b8633bf8e5a4dc13f58d59cc4a3c52901231a484a1
Sep 10 17:23:53.083900 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.083869    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfc98be_bcb2_44bd_a4eb_fe5cab6d6571.slice/crio-71a5c9dc9a7cefadeffb067e202884553e04036b9316e43f8b77875ce6daa5d0 WatchSource:0}: Error finding container 71a5c9dc9a7cefadeffb067e202884553e04036b9316e43f8b77875ce6daa5d0: Status 404 returned error can't find the container with id 71a5c9dc9a7cefadeffb067e202884553e04036b9316e43f8b77875ce6daa5d0
Sep 10 17:23:53.085582 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.085563    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5r7kc"
Sep 10 17:23:53.093208 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.093189    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pd8gz"
Sep 10 17:23:53.093997 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.093980    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78220b5d_9112_416d_a65f_c2c7f3359167.slice/crio-e9da72505b3aae35967b9a5ed41899793f32cf177d13b77a65735ea70242ff68 WatchSource:0}: Error finding container e9da72505b3aae35967b9a5ed41899793f32cf177d13b77a65735ea70242ff68: Status 404 returned error can't find the container with id e9da72505b3aae35967b9a5ed41899793f32cf177d13b77a65735ea70242ff68
Sep 10 17:23:53.098128 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.098113    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2rtwg"
Sep 10 17:23:53.102397 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.102378    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9038897d_9736_4d81_a7b8_ea5b7d52e587.slice/crio-b7065eb6df1f70ddec8f1fc1f272ca3648bea5049e8383c429a41bc0b650b9b1 WatchSource:0}: Error finding container b7065eb6df1f70ddec8f1fc1f272ca3648bea5049e8383c429a41bc0b650b9b1: Status 404 returned error can't find the container with id b7065eb6df1f70ddec8f1fc1f272ca3648bea5049e8383c429a41bc0b650b9b1
Sep 10 17:23:53.106929 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:23:53.106910    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e68c07_a8f9_4182_a40c_a1ef1470e7cc.slice/crio-4eeabc1bfa0bf0ec21701d1f7d2412d1f43a3176430f1af23b04e1a54c159658 WatchSource:0}: Error finding container 4eeabc1bfa0bf0ec21701d1f7d2412d1f43a3176430f1af23b04e1a54c159658: Status 404 returned error can't find the container with id 4eeabc1bfa0bf0ec21701d1f7d2412d1f43a3176430f1af23b04e1a54c159658
Sep 10 17:23:53.379206 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.379120    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:53.379206 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.379181    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:53.379449 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:53.379324    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:53.379449 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:53.379370    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:53.379449 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:53.379403    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret podName:ced14a44-5fd5-4179-af31-eef5392a3e7c nodeName:}" failed. No retries permitted until 2026-09-10 17:23:54.379383127 +0000 UTC m=+3.088977399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret") pod "global-pull-secret-syncer-5gjsv" (UID: "ced14a44-5fd5-4179-af31-eef5392a3e7c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:53.379449 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:53.379424    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs podName:3ea78e61-4de5-43fa-81a2-8ffff13e1d3a nodeName:}" failed. No retries permitted until 2026-09-10 17:23:54.379413319 +0000 UTC m=+3.089007578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs") pod "network-metrics-daemon-gl4v7" (UID: "3ea78e61-4de5-43fa-81a2-8ffff13e1d3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:53.479673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.479632    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:53.479883 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:53.479865    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:53.479952 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:53.479889    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:53.479952 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:53.479903    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lh8nc for pod openshift-network-diagnostics/network-check-target-njlj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:53.480054 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:53.479962    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc podName:cbade337-a7dc-4cab-9479-2303873983b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:54.479943528 +0000 UTC m=+3.189537793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8nc" (UniqueName: "kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc") pod "network-check-target-njlj5" (UID: "cbade337-a7dc-4cab-9479-2303873983b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:53.653353 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.653065    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:53.749387 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.749253    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:18:53 +0000 UTC" deadline="2028-06-05 07:47:51.990784054 +0000 UTC"
Sep 10 17:23:53.749387 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.749317    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15206h23m58.241471497s"
Sep 10 17:23:53.770096 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.770031    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2rtwg" event={"ID":"63e68c07-a8f9-4182-a40c-a1ef1470e7cc","Type":"ContainerStarted","Data":"4eeabc1bfa0bf0ec21701d1f7d2412d1f43a3176430f1af23b04e1a54c159658"}
Sep 10 17:23:53.779329 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.779287    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pntqj" event={"ID":"8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571","Type":"ContainerStarted","Data":"71a5c9dc9a7cefadeffb067e202884553e04036b9316e43f8b77875ce6daa5d0"}
Sep 10 17:23:53.792293 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.792258    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" event={"ID":"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7","Type":"ContainerStarted","Data":"17bebb4940e3b13246a412b8633bf8e5a4dc13f58d59cc4a3c52901231a484a1"}
Sep 10 17:23:53.810853 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.810710    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kglhk" event={"ID":"d080ad12-87b4-483f-b0b8-5ab978b22216","Type":"ContainerStarted","Data":"e03272a77ae82929a6451cb2eb688bcf8471fbdaef5f1141a82ac92544c40a9e"}
Sep 10 17:23:53.831395 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.831207    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57" event={"ID":"39d4f881-f1f5-48ac-ae30-401b708297d2","Type":"ContainerStarted","Data":"b55bf2324aa9c4226ba58741d2089c0d084648b814314edd08ca235e6a2a8caa"}
Sep 10 17:23:53.833022 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.832993    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal" event={"ID":"7091a140b9b7cb44dee6eef96e11a133","Type":"ContainerStarted","Data":"d688ae64e67ca9ad81fe49c7f3c4e50bfc85fec41b4f3d75a3c7d5d2b81f38b7"}
Sep 10 17:23:53.836534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.836508    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pd8gz" event={"ID":"9038897d-9736-4d81-a7b8-ea5b7d52e587","Type":"ContainerStarted","Data":"b7065eb6df1f70ddec8f1fc1f272ca3648bea5049e8383c429a41bc0b650b9b1"}
Sep 10 17:23:53.852141 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.852101    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5r7kc" event={"ID":"78220b5d-9112-416d-a65f-c2c7f3359167","Type":"ContainerStarted","Data":"e9da72505b3aae35967b9a5ed41899793f32cf177d13b77a65735ea70242ff68"}
Sep 10 17:23:53.862533 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.862504    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2pdpk" event={"ID":"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54","Type":"ContainerStarted","Data":"7ea674e42035b202b8254921868108b629ee5043ce7daf6f4ee5eec94effbdb2"}
Sep 10 17:23:53.872707 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.872676    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"7cced750df23224789083193f8453dfdaf44b1cb13faf30f9b8010d68d5eafbe"}
Sep 10 17:23:53.879034 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:53.879005    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal" event={"ID":"1f3333ec3c604b826ba36ee4849cfacb","Type":"ContainerStarted","Data":"d48a92e5f867c899bb93dadd64e76f1b4e56527bc6e57752033b46e267e6eee6"}
Sep 10 17:23:54.386545 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.386502    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:54.386731 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.386571    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:54.386793 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.386732    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:54.386966 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.386946    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:54.387048 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.387010    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret podName:ced14a44-5fd5-4179-af31-eef5392a3e7c nodeName:}" failed. No retries permitted until 2026-09-10 17:23:56.386990493 +0000 UTC m=+5.096584743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret") pod "global-pull-secret-syncer-5gjsv" (UID: "ced14a44-5fd5-4179-af31-eef5392a3e7c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:54.387126 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.387069    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs podName:3ea78e61-4de5-43fa-81a2-8ffff13e1d3a nodeName:}" failed. No retries permitted until 2026-09-10 17:23:56.387054205 +0000 UTC m=+5.096648456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs") pod "network-metrics-daemon-gl4v7" (UID: "3ea78e61-4de5-43fa-81a2-8ffff13e1d3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:54.487973 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.487307    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:54.487973 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.487509    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:54.487973 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.487531    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:54.487973 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.487545    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lh8nc for pod openshift-network-diagnostics/network-check-target-njlj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:54.487973 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.487609    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc podName:cbade337-a7dc-4cab-9479-2303873983b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:23:56.487588136 +0000 UTC m=+5.197182384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8nc" (UniqueName: "kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc") pod "network-check-target-njlj5" (UID: "cbade337-a7dc-4cab-9479-2303873983b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:54.749638 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.749438    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:18:53 +0000 UTC" deadline="2028-05-02 21:19:47.857744761 +0000 UTC"
Sep 10 17:23:54.749638 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.749515    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14403h55m53.108234055s"
Sep 10 17:23:54.758578 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.757663    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:54.758578 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.757815    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:23:54.758578 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.758263    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:54.758578 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.758381    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:23:54.758578 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.758447    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:54.758578 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:54.758521    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:23:54.773223 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:54.773004    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:23:56.405904 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:56.405821    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:56.405904 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:56.405879    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:56.406640 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.406008    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:56.406640 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.406070    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs podName:3ea78e61-4de5-43fa-81a2-8ffff13e1d3a nodeName:}" failed. No retries permitted until 2026-09-10 17:24:00.406050969 +0000 UTC m=+9.115645223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs") pod "network-metrics-daemon-gl4v7" (UID: "3ea78e61-4de5-43fa-81a2-8ffff13e1d3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:23:56.406640 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.406494    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:56.406640 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.406544    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret podName:ced14a44-5fd5-4179-af31-eef5392a3e7c nodeName:}" failed. No retries permitted until 2026-09-10 17:24:00.406528907 +0000 UTC m=+9.116123160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret") pod "global-pull-secret-syncer-5gjsv" (UID: "ced14a44-5fd5-4179-af31-eef5392a3e7c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:23:56.506969 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:56.506838    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:56.507145 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.507002    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:23:56.507145 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.507040    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:23:56.507145 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.507054    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lh8nc for pod openshift-network-diagnostics/network-check-target-njlj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:56.507145 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.507119    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc podName:cbade337-a7dc-4cab-9479-2303873983b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:00.507099075 +0000 UTC m=+9.216693328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8nc" (UniqueName: "kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc") pod "network-check-target-njlj5" (UID: "cbade337-a7dc-4cab-9479-2303873983b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:23:56.758203 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:56.757693    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:56.758203 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:56.757732    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:56.758203 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:56.757732    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:56.758203 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.757833    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:23:56.758203 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.757949    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:23:56.758203 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:56.758042    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:23:58.758106 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:58.758061    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:23:58.758625 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:58.758220    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:23:58.758625 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:58.758277    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:23:58.758625 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:23:58.758290    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:23:58.758625 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:58.758363    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:23:58.758625 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:23:58.758459    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:00.442033 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:00.441996    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:00.442484 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:00.442052    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:00.442484 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.442222    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:00.442484 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.442250    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:00.442484 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.442318    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs podName:3ea78e61-4de5-43fa-81a2-8ffff13e1d3a nodeName:}" failed. No retries permitted until 2026-09-10 17:24:08.442277863 +0000 UTC m=+17.151872112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs") pod "network-metrics-daemon-gl4v7" (UID: "3ea78e61-4de5-43fa-81a2-8ffff13e1d3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:00.442484 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.442340    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret podName:ced14a44-5fd5-4179-af31-eef5392a3e7c nodeName:}" failed. No retries permitted until 2026-09-10 17:24:08.442329247 +0000 UTC m=+17.151923501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret") pod "global-pull-secret-syncer-5gjsv" (UID: "ced14a44-5fd5-4179-af31-eef5392a3e7c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:00.542631 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:00.542552    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:00.542815 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.542757    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:24:00.542815 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.542795    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:24:00.542815 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.542809    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lh8nc for pod openshift-network-diagnostics/network-check-target-njlj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:00.543115 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.542881    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc podName:cbade337-a7dc-4cab-9479-2303873983b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:08.542858979 +0000 UTC m=+17.252453267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8nc" (UniqueName: "kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc") pod "network-check-target-njlj5" (UID: "cbade337-a7dc-4cab-9479-2303873983b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:00.758456 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:00.757563    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:00.758456 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.757728    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:00.758456 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:00.758019    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:00.758456 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.758129    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:00.758456 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:00.758158    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:00.758456 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:00.758284    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:02.757728 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:02.757695    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:02.758151 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:02.757820    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:02.758151 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:02.757901    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:02.758151 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:02.758035    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:02.758151 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:02.758032    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:02.758151 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:02.758106    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:04.757583 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:04.757539    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:04.758061 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:04.757539    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:04.758061 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:04.757637    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:04.758061 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:04.757638    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:04.758061 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:04.757706    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:04.758061 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:04.757763    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:06.757679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:06.757638    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:06.757679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:06.757677    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:06.758143 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:06.757638    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:06.758143 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:06.757777    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:06.758143 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:06.757844    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:06.758143 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:06.757946    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:08.505810 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:08.505771    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:08.506219 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:08.505832    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:08.506219 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.505938    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:08.506219 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.506000    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs podName:3ea78e61-4de5-43fa-81a2-8ffff13e1d3a nodeName:}" failed. No retries permitted until 2026-09-10 17:24:24.505980684 +0000 UTC m=+33.215574946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs") pod "network-metrics-daemon-gl4v7" (UID: "3ea78e61-4de5-43fa-81a2-8ffff13e1d3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:08.506219 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.506076    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:08.506219 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.506144    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret podName:ced14a44-5fd5-4179-af31-eef5392a3e7c nodeName:}" failed. No retries permitted until 2026-09-10 17:24:24.506126786 +0000 UTC m=+33.215721046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret") pod "global-pull-secret-syncer-5gjsv" (UID: "ced14a44-5fd5-4179-af31-eef5392a3e7c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:08.606400 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:08.606359    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:08.606615 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.606512    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:24:08.606615 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.606534    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:24:08.606615 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.606546    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lh8nc for pod openshift-network-diagnostics/network-check-target-njlj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:08.606615 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.606609    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc podName:cbade337-a7dc-4cab-9479-2303873983b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:24.606591674 +0000 UTC m=+33.316185927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8nc" (UniqueName: "kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc") pod "network-check-target-njlj5" (UID: "cbade337-a7dc-4cab-9479-2303873983b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:08.757188 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:08.757092    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:08.757188 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:08.757134    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:08.757419 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.757257    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:08.757419 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.757314    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:08.757419 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:08.757367    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:08.757556 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:08.757435    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:10.757921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:10.757886    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:10.757921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:10.757919    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:10.758409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:10.757931    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:10.758409 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:10.758040    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:10.758409 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:10.758178    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:10.758409 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:10.758276    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:12.757081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.757055    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:12.757440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.757083    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:12.757440 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:12.757165    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:12.757440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.757217    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:12.757440 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:12.757321    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:12.757440 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:12.757391    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:12.925982 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.925950    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal" event={"ID":"1f3333ec3c604b826ba36ee4849cfacb","Type":"ContainerStarted","Data":"4176d59e918840d3aad0172d9e3dc577bc143b29eca662cefed8dd1f8e8bd3b7"}
Sep 10 17:24:12.927690 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.927662    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5r7kc" event={"ID":"78220b5d-9112-416d-a65f-c2c7f3359167","Type":"ContainerStarted","Data":"07ab5b3a2811ddf1c400e5bf9d86b0073ef11bffb7c98bd3ea0afb52a7614365"}
Sep 10 17:24:12.931753 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.931725    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"f5e59d3d369766c81bf3c9b680f1cfb24ba480e31ce4201be8ff6f3186366e0b"}
Sep 10 17:24:12.931847 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.931807    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"412cc1460e6fac6501113991966af7deb3313cdce7b5894dd694fad3b973201d"}
Sep 10 17:24:12.941443 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.941380    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-242.ec2.internal" podStartSLOduration=19.941360108 podStartE2EDuration="19.941360108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:12.941136767 +0000 UTC m=+21.650731057" watchObservedRunningTime="2026-09-10 17:24:12.941360108 +0000 UTC m=+21.650954374"
Sep 10 17:24:12.961565 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:12.961517    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5r7kc" podStartSLOduration=1.570520219 podStartE2EDuration="20.961496472s" totalImagesPullingTime="19.390976253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:12.961153749 +0000 UTC m=+21.670748015" watchObservedRunningTime="2026-09-10 17:24:12.961496472 +0000 UTC m=+21.671090745"
Sep 10 17:24:13.937325 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.937297    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:24:13.938167 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.937665    2568 generic.go:356] "Generic (PLEG): container finished" podID="f7bc1e14-5460-41d2-a808-8ebb92f3c29f" containerID="f5e59d3d369766c81bf3c9b680f1cfb24ba480e31ce4201be8ff6f3186366e0b" exitCode=1
Sep 10 17:24:13.938167 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.937784    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerDied","Data":"f5e59d3d369766c81bf3c9b680f1cfb24ba480e31ce4201be8ff6f3186366e0b"}
Sep 10 17:24:13.938167 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.937882    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"98940af2768d4b8de07979c6716ea239f014c9b3bae50490e8caffb24d51295e"}
Sep 10 17:24:13.938167 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.937935    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"e89eb6ecaa3031638fcefe688060134e402d76056ef7e25686e9915e3f3389b9"}
Sep 10 17:24:13.938167 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.937983    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"8168ede52df07fe353a3e7d527877687c4a62cf0116a76e1717de64f13a85ac0"}
Sep 10 17:24:13.938167 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.938031    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"b551760ef7b77e4192dd1502e8e2934f4d8c1682cfbee93509869b6da042707c"}
Sep 10 17:24:13.939656 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.939628    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2rtwg" event={"ID":"63e68c07-a8f9-4182-a40c-a1ef1470e7cc","Type":"ContainerStarted","Data":"3486beb1dc2d04e17963eaccf4876bb61501b63c6ff55d3fb124a2e7e74179e0"}
Sep 10 17:24:13.941272 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.941229    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pntqj" event={"ID":"8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571","Type":"ContainerStarted","Data":"8a87a4a1c7707a274ee1bcab9b1564e0ef063e8c0ba401c6b50ba5ae0d718d70"}
Sep 10 17:24:13.943189 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.943143    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7" containerID="b9e9602ad4e77e497bc6221a0bab2d78ce8875e66090f6e49dff5e2bdbb5a9cc" exitCode=0
Sep 10 17:24:13.943306 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.943243    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" event={"ID":"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7","Type":"ContainerDied","Data":"b9e9602ad4e77e497bc6221a0bab2d78ce8875e66090f6e49dff5e2bdbb5a9cc"}
Sep 10 17:24:13.945723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.945696    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57" event={"ID":"39d4f881-f1f5-48ac-ae30-401b708297d2","Type":"ContainerStarted","Data":"4982ea2a576b27e6b324c51ecc8882aeb5f4ed63b24f6f545cd12338b5710b03"}
Sep 10 17:24:13.947455 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.947433    2568 generic.go:356] "Generic (PLEG): container finished" podID="7091a140b9b7cb44dee6eef96e11a133" containerID="0828d2725d50c6ad5b3f19dfc3c2de6600e2b92a523e49cb7f994d2187916bf2" exitCode=0
Sep 10 17:24:13.947676 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.947655    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal" event={"ID":"7091a140b9b7cb44dee6eef96e11a133","Type":"ContainerDied","Data":"0828d2725d50c6ad5b3f19dfc3c2de6600e2b92a523e49cb7f994d2187916bf2"}
Sep 10 17:24:13.949286 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.949260    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pd8gz" event={"ID":"9038897d-9736-4d81-a7b8-ea5b7d52e587","Type":"ContainerStarted","Data":"5d4fd95cb8478eb778ebf8200c026cf576b45196757b670824655f1531c066ab"}
Sep 10 17:24:13.951031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.951002    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2pdpk" event={"ID":"f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54","Type":"ContainerStarted","Data":"a44052bfa1877fb0fe0553c47421cf72b8b9612c347ed2e4afdbbc6bc672c0ed"}
Sep 10 17:24:13.958931 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:13.958890    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2rtwg" podStartSLOduration=2.630941202 podStartE2EDuration="21.958879431s" totalImagesPullingTime="19.327938229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:13.958258932 +0000 UTC m=+22.667853196" watchObservedRunningTime="2026-09-10 17:24:13.958879431 +0000 UTC m=+22.668473711"
Sep 10 17:24:14.007931 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.007880    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2pdpk" podStartSLOduration=2.591723177 podStartE2EDuration="22.00785972s" totalImagesPullingTime="19.416136543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:14.007696188 +0000 UTC m=+22.717290455" watchObservedRunningTime="2026-09-10 17:24:14.00785972 +0000 UTC m=+22.717453988"
Sep 10 17:24:14.022390 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.022346    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pntqj" podStartSLOduration=6.655745447 podStartE2EDuration="22.022328587s" totalImagesPullingTime="15.36658314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:14.021707359 +0000 UTC m=+22.731301627" watchObservedRunningTime="2026-09-10 17:24:14.022328587 +0000 UTC m=+22.731922856"
Sep 10 17:24:14.041384 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.041158    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pd8gz" podStartSLOduration=2.06355443 podStartE2EDuration="22.04114425s" totalImagesPullingTime="19.97758982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:14.040491614 +0000 UTC m=+22.750085883" watchObservedRunningTime="2026-09-10 17:24:14.04114425 +0000 UTC m=+22.750738521"
Sep 10 17:24:14.456300 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.456274    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:24:14.755109 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.755085    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 17:24:14.757201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.757185    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:14.757329 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.757310    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:14.757329 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:14.757319    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:14.757403 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:14.757388    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:14.757464 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.757454    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:14.757517 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:14.757505    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:14.955282 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.955222    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kglhk" event={"ID":"d080ad12-87b4-483f-b0b8-5ab978b22216","Type":"ContainerStarted","Data":"4f5b61d17a49498be59d78f30b195827397bae54c881e1223ec7314ac621b61b"}
Sep 10 17:24:14.957761 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.957686    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57" event={"ID":"39d4f881-f1f5-48ac-ae30-401b708297d2","Type":"ContainerStarted","Data":"d2e65432bccc0c6c30eeb066ee5b8d0b1a276be679a13c4b9fa4cd81a18d631d"}
Sep 10 17:24:14.959869 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.959832    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal" event={"ID":"7091a140b9b7cb44dee6eef96e11a133","Type":"ContainerStarted","Data":"3b10ce2756b69dc9cccd39fe58110d2d1d83f90f4a61b76003b42c338dbe6d04"}
Sep 10 17:24:14.970167 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.970121    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kglhk" podStartSLOduration=3.535870031 podStartE2EDuration="22.970104209s" totalImagesPullingTime="19.434234178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:14.96970296 +0000 UTC m=+23.679297227" watchObservedRunningTime="2026-09-10 17:24:14.970104209 +0000 UTC m=+23.679698475"
Sep 10 17:24:14.985015 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:14.984968    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-242.ec2.internal" podStartSLOduration=21.98495639 podStartE2EDuration="21.98495639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:14.984649253 +0000 UTC m=+23.694243520" watchObservedRunningTime="2026-09-10 17:24:14.98495639 +0000 UTC m=+23.694550655"
Sep 10 17:24:15.727428 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:15.727254    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T17:24:14.755104113Z","UUID":"60352fda-6e4e-43c1-82b4-fd6a5087107b","Handler":null,"Name":"","Endpoint":""}
Sep 10 17:24:15.728909 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:15.728884    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 17:24:15.729017 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:15.728923    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 17:24:15.966033 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:15.966009    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:24:15.966545 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:15.966512    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"a38d44d8739fdca0c76ef93bf86dab142fce2489b76800c4888c850ec616145b"}
Sep 10 17:24:16.757756 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:16.757727    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:16.757756 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:16.757755    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:16.758009 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:16.757864    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:16.758009 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:16.757908    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:16.758009 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:16.757960    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:16.758124 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:16.758035    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:16.971780 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:16.971744    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57" event={"ID":"39d4f881-f1f5-48ac-ae30-401b708297d2","Type":"ContainerStarted","Data":"4f3907d3799b6b4e6871791a5db0d1058a59d8895a4ea3c8f42c1f8026cd7f3f"}
Sep 10 17:24:16.991613 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:16.991549    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xl57" podStartSLOduration=3.150646112 podStartE2EDuration="24.99153368s" totalImagesPullingTime="21.840887568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:16.991020956 +0000 UTC m=+25.700615221" watchObservedRunningTime="2026-09-10 17:24:16.99153368 +0000 UTC m=+25.701127977"
Sep 10 17:24:18.758117 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.757880    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:18.758446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.757881    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:18.758446 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:18.758192    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:18.758446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.757952    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:18.758446 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:18.758272    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:18.758446 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:18.758330    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:18.940203 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.940177    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:24:18.940788 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.940767    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:24:18.980860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.980836    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:24:18.981323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.981290    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"9198607a44712ce05b2616be4e00e924cc94747cdd9a1d98045a60b12856bb2d"}
Sep 10 17:24:18.981832 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.981809    2568 scope.go:117] "RemoveContainer" containerID="f5e59d3d369766c81bf3c9b680f1cfb24ba480e31ce4201be8ff6f3186366e0b"
Sep 10 17:24:18.982120 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:18.982104    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pntqj"
Sep 10 17:24:19.989760 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:19.989672    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:24:19.990669 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:19.990561    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" event={"ID":"f7bc1e14-5460-41d2-a808-8ebb92f3c29f","Type":"ContainerStarted","Data":"c8c3a8305835548c32dbe84132ae10f5953eb3b88820b957c0804d32adedb64b"}
Sep 10 17:24:19.991876 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:19.991845    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:24:19.992002 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:19.991941    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:24:19.992059 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:19.992052    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:24:19.998790 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:19.996512    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7" containerID="c3888195b575fb7809ee8bd4c68a244eecc5e9d22a146e63e847bcc0aa928ec6" exitCode=0
Sep 10 17:24:19.998790 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:19.997706    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" event={"ID":"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7","Type":"ContainerDied","Data":"c3888195b575fb7809ee8bd4c68a244eecc5e9d22a146e63e847bcc0aa928ec6"}
Sep 10 17:24:20.025548 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.025493    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh" podStartSLOduration=8.533363407 podStartE2EDuration="28.025477565s" totalImagesPullingTime="19.492114158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:20.025145658 +0000 UTC m=+28.734739924" watchObservedRunningTime="2026-09-10 17:24:20.025477565 +0000 UTC m=+28.735071831"
Sep 10 17:24:20.026760 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.026586    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:24:20.029487 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.029468    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:24:20.615905 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.615681    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gl4v7"]
Sep 10 17:24:20.616219 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.616034    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:20.616219 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:20.616164    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:20.617770 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.617748    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njlj5"]
Sep 10 17:24:20.617900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.617837    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:20.617959 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:20.617915    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:20.620411 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.620388    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5gjsv"]
Sep 10 17:24:20.620523 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:20.620489    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:20.620605 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:20.620582    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:22.003283 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:22.003187    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7" containerID="a29117da89b3449a3f22f9e4c33db4913fdbe2426d4db5c6eabd7fe3f11ad993" exitCode=0
Sep 10 17:24:22.003645 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:22.003270    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" event={"ID":"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7","Type":"ContainerDied","Data":"a29117da89b3449a3f22f9e4c33db4913fdbe2426d4db5c6eabd7fe3f11ad993"}
Sep 10 17:24:22.757251 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:22.757078    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:22.757350 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:22.757077    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:22.757391 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:22.757341    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:22.757436 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:22.757400    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:22.757436 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:22.757132    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:22.757507 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:22.757475    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:23.007939 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:23.007859    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7" containerID="42b0a803979bb91980549b18e43dd5a65722d2d58df03c7f2bbcef74133022d1" exitCode=0
Sep 10 17:24:23.007939 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:23.007921    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" event={"ID":"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7","Type":"ContainerDied","Data":"42b0a803979bb91980549b18e43dd5a65722d2d58df03c7f2bbcef74133022d1"}
Sep 10 17:24:24.517549 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:24.517510    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:24.518001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:24.517564    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:24.518001 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.517680    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:24.518001 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.517743    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:24.518001 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.517758    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret podName:ced14a44-5fd5-4179-af31-eef5392a3e7c nodeName:}" failed. No retries permitted until 2026-09-10 17:24:56.51773805 +0000 UTC m=+65.227332298 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret") pod "global-pull-secret-syncer-5gjsv" (UID: "ced14a44-5fd5-4179-af31-eef5392a3e7c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:24:24.518001 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.517784    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs podName:3ea78e61-4de5-43fa-81a2-8ffff13e1d3a nodeName:}" failed. No retries permitted until 2026-09-10 17:24:56.517771777 +0000 UTC m=+65.227366021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs") pod "network-metrics-daemon-gl4v7" (UID: "3ea78e61-4de5-43fa-81a2-8ffff13e1d3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:24:24.618307 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:24.618273    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:24.618523 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.618448    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:24:24.618523 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.618468    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:24:24.618523 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.618477    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lh8nc for pod openshift-network-diagnostics/network-check-target-njlj5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:24.618647 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.618541    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc podName:cbade337-a7dc-4cab-9479-2303873983b3 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:56.618520159 +0000 UTC m=+65.328114418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8nc" (UniqueName: "kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc") pod "network-check-target-njlj5" (UID: "cbade337-a7dc-4cab-9479-2303873983b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:24:24.757151 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:24.757117    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:24.757353 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:24.757116    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:24.757353 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:24.757117    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:24.757353 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.757283    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gl4v7" podUID="3ea78e61-4de5-43fa-81a2-8ffff13e1d3a"
Sep 10 17:24:24.757508 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.757374    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njlj5" podUID="cbade337-a7dc-4cab-9479-2303873983b3"
Sep 10 17:24:24.757508 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:24.757453    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gjsv" podUID="ced14a44-5fd5-4179-af31-eef5392a3e7c"
Sep 10 17:24:25.623204 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.623177    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-242.ec2.internal" event="NodeReady"
Sep 10 17:24:25.623663 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.623353    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 17:24:25.675811 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.675700    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8lqdv"]
Sep 10 17:24:25.682515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.682492    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bhlnf"]
Sep 10 17:24:25.682671 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.682630    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.685341 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.685320    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vn8bp\""
Sep 10 17:24:25.685496 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.685465    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 17:24:25.685711 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.685696    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 17:24:25.688581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.688543    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhlnf"
Sep 10 17:24:25.690992 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.690969    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 17:24:25.691091 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.691053    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dfch6\""
Sep 10 17:24:25.691091 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.691073    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 17:24:25.691387 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.691327    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 17:24:25.694350 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.694214    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lqdv"]
Sep 10 17:24:25.695004 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.694984    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhlnf"]
Sep 10 17:24:25.825660 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.825626    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cljk\" (UniqueName: \"kubernetes.io/projected/412fe3b2-c273-4b82-b8a9-cf0559396157-kube-api-access-9cljk\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.825846 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.825683    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/945e40c2-0163-4303-98ca-1519c942c638-cert\") pod \"ingress-canary-bhlnf\" (UID: \"945e40c2-0163-4303-98ca-1519c942c638\") " pod="openshift-ingress-canary/ingress-canary-bhlnf"
Sep 10 17:24:25.825846 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.825710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.825846 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.825745    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/412fe3b2-c273-4b82-b8a9-cf0559396157-config-volume\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.826002 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.825872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txntt\" (UniqueName: \"kubernetes.io/projected/945e40c2-0163-4303-98ca-1519c942c638-kube-api-access-txntt\") pod \"ingress-canary-bhlnf\" (UID: \"945e40c2-0163-4303-98ca-1519c942c638\") " pod="openshift-ingress-canary/ingress-canary-bhlnf"
Sep 10 17:24:25.826002 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.825901    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/412fe3b2-c273-4b82-b8a9-cf0559396157-tmp-dir\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.926936 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.926896    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txntt\" (UniqueName: \"kubernetes.io/projected/945e40c2-0163-4303-98ca-1519c942c638-kube-api-access-txntt\") pod \"ingress-canary-bhlnf\" (UID: \"945e40c2-0163-4303-98ca-1519c942c638\") " pod="openshift-ingress-canary/ingress-canary-bhlnf"
Sep 10 17:24:25.927111 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.926948    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/412fe3b2-c273-4b82-b8a9-cf0559396157-tmp-dir\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.927111 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.926977    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cljk\" (UniqueName: \"kubernetes.io/projected/412fe3b2-c273-4b82-b8a9-cf0559396157-kube-api-access-9cljk\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.927111 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.927004    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/945e40c2-0163-4303-98ca-1519c942c638-cert\") pod \"ingress-canary-bhlnf\" (UID: \"945e40c2-0163-4303-98ca-1519c942c638\") " pod="openshift-ingress-canary/ingress-canary-bhlnf"
Sep 10 17:24:25.927111 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.927019    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.927111 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.927051    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/412fe3b2-c273-4b82-b8a9-cf0559396157-config-volume\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.927395 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:25.927287    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:25.927395 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:25.927355    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls podName:412fe3b2-c273-4b82-b8a9-cf0559396157 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:26.427334708 +0000 UTC m=+35.136928969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls") pod "dns-default-8lqdv" (UID: "412fe3b2-c273-4b82-b8a9-cf0559396157") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:25.927664 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.927633    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/412fe3b2-c273-4b82-b8a9-cf0559396157-tmp-dir\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.927792 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.927708    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/412fe3b2-c273-4b82-b8a9-cf0559396157-config-volume\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:25.931767 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.931749    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/945e40c2-0163-4303-98ca-1519c942c638-cert\") pod \"ingress-canary-bhlnf\" (UID: \"945e40c2-0163-4303-98ca-1519c942c638\") " pod="openshift-ingress-canary/ingress-canary-bhlnf"
Sep 10 17:24:25.936249 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.936202    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txntt\" (UniqueName: \"kubernetes.io/projected/945e40c2-0163-4303-98ca-1519c942c638-kube-api-access-txntt\") pod \"ingress-canary-bhlnf\" (UID: \"945e40c2-0163-4303-98ca-1519c942c638\") " pod="openshift-ingress-canary/ingress-canary-bhlnf"
Sep 10 17:24:25.936682 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:25.936663    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cljk\" (UniqueName: \"kubernetes.io/projected/412fe3b2-c273-4b82-b8a9-cf0559396157-kube-api-access-9cljk\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:26.002367 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.002321    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhlnf"
Sep 10 17:24:26.430336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.430293    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:26.430519 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:26.430480    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:26.430574 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:26.430563    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls podName:412fe3b2-c273-4b82-b8a9-cf0559396157 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:27.430546269 +0000 UTC m=+36.140140537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls") pod "dns-default-8lqdv" (UID: "412fe3b2-c273-4b82-b8a9-cf0559396157") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:26.758194 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.758106    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:26.758194 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.758142    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:26.758194 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.758146    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:26.761583 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.761558    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:24:26.761705 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.761601    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:24:26.762839 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.762821    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:24:26.762973 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.762826    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qjn9k\""
Sep 10 17:24:26.762973 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.762919    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v58r5\""
Sep 10 17:24:26.762973 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:26.762936    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:24:27.341364 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:27.341291    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhlnf"]
Sep 10 17:24:27.344596 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:24:27.344561    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945e40c2_0163_4303_98ca_1519c942c638.slice/crio-27065228b9103845e6c6346d9c8115daceac99533ac884b729d1c28c5ead3429 WatchSource:0}: Error finding container 27065228b9103845e6c6346d9c8115daceac99533ac884b729d1c28c5ead3429: Status 404 returned error can't find the container with id 27065228b9103845e6c6346d9c8115daceac99533ac884b729d1c28c5ead3429
Sep 10 17:24:27.438188 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:27.438157    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:27.438332 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:27.438299    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:27.438378 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:27.438358    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls podName:412fe3b2-c273-4b82-b8a9-cf0559396157 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:29.438345693 +0000 UTC m=+38.147939951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls") pod "dns-default-8lqdv" (UID: "412fe3b2-c273-4b82-b8a9-cf0559396157") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:28.022066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:28.021852    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7" containerID="25aafb41aca44691bc0cb065d477deeea34c69608c15cb751c0b7a5dfc54c95f" exitCode=0
Sep 10 17:24:28.022758 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:28.021919    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" event={"ID":"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7","Type":"ContainerDied","Data":"25aafb41aca44691bc0cb065d477deeea34c69608c15cb751c0b7a5dfc54c95f"}
Sep 10 17:24:28.023429 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:28.023404    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhlnf" event={"ID":"945e40c2-0163-4303-98ca-1519c942c638","Type":"ContainerStarted","Data":"27065228b9103845e6c6346d9c8115daceac99533ac884b729d1c28c5ead3429"}
Sep 10 17:24:29.029321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:29.029282    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7" containerID="d99518a083476883ebff9c260a9962b1b4336bb85a80d7028c54793e3f890704" exitCode=0
Sep 10 17:24:29.029784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:29.029366    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" event={"ID":"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7","Type":"ContainerDied","Data":"d99518a083476883ebff9c260a9962b1b4336bb85a80d7028c54793e3f890704"}
Sep 10 17:24:29.451859 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:29.451783    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:29.451993 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:29.451908    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:29.451993 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:29.451964    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls podName:412fe3b2-c273-4b82-b8a9-cf0559396157 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:33.45194888 +0000 UTC m=+42.161543124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls") pod "dns-default-8lqdv" (UID: "412fe3b2-c273-4b82-b8a9-cf0559396157") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:30.034822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:30.034788    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" event={"ID":"5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7","Type":"ContainerStarted","Data":"79bfdc1e978301069b13363db8e11dcb0700cf445d2ca3806b95733b4db7a2b5"}
Sep 10 17:24:30.059013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:30.058958    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2r8ch" podStartSLOduration=7.671024401 podStartE2EDuration="38.058944063s" totalImagesPullingTime="30.387919662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:30.057486885 +0000 UTC m=+38.767081165" watchObservedRunningTime="2026-09-10 17:24:30.058944063 +0000 UTC m=+38.768538329"
Sep 10 17:24:31.038584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:31.038541    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhlnf" event={"ID":"945e40c2-0163-4303-98ca-1519c942c638","Type":"ContainerStarted","Data":"05ab3cff7295ef3565b61b71a4bf1b05ba11c56ad9aa5bb749bc17dab761e99e"}
Sep 10 17:24:31.059303 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:31.059255    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bhlnf" podStartSLOduration=3.458433583 podStartE2EDuration="6.059220417s" totalImagesPullingTime="2.600786834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:31.057927579 +0000 UTC m=+39.767521844" watchObservedRunningTime="2026-09-10 17:24:31.059220417 +0000 UTC m=+39.768814682"
Sep 10 17:24:33.481858 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:33.481815    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:33.482294 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:33.481972    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:33.482294 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:33.482053    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls podName:412fe3b2-c273-4b82-b8a9-cf0559396157 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:41.482036179 +0000 UTC m=+50.191630438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls") pod "dns-default-8lqdv" (UID: "412fe3b2-c273-4b82-b8a9-cf0559396157") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:41.537486 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:41.537449    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:41.537923 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:41.537563    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:41.537923 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:41.537616    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls podName:412fe3b2-c273-4b82-b8a9-cf0559396157 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:57.537602121 +0000 UTC m=+66.247196365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls") pod "dns-default-8lqdv" (UID: "412fe3b2-c273-4b82-b8a9-cf0559396157") : secret "dns-default-metrics-tls" not found
Sep 10 17:24:52.013803 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:52.013774    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-znlbh"
Sep 10 17:24:56.541389 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.541341    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:56.541389 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.541397    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:24:56.547126 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.547101    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:24:56.549858 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.549842    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:24:56.551728 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:56.551712    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 17:24:56.551811 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:56.551789    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs podName:3ea78e61-4de5-43fa-81a2-8ffff13e1d3a nodeName:}" failed. No retries permitted until 2026-09-10 17:26:00.551767531 +0000 UTC m=+129.261361775 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs") pod "network-metrics-daemon-gl4v7" (UID: "3ea78e61-4de5-43fa-81a2-8ffff13e1d3a") : secret "metrics-daemon-secret" not found
Sep 10 17:24:56.555017 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.555000    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced14a44-5fd5-4179-af31-eef5392a3e7c-original-pull-secret\") pod \"global-pull-secret-syncer-5gjsv\" (UID: \"ced14a44-5fd5-4179-af31-eef5392a3e7c\") " pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:56.641992 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.641951    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:56.644965 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.644941    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:24:56.654474 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.654455    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:24:56.664900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.664873    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8nc\" (UniqueName: \"kubernetes.io/projected/cbade337-a7dc-4cab-9479-2303873983b3-kube-api-access-lh8nc\") pod \"network-check-target-njlj5\" (UID: \"cbade337-a7dc-4cab-9479-2303873983b3\") " pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:56.770009 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.769978    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gjsv"
Sep 10 17:24:56.784101 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.784076    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v58r5\""
Sep 10 17:24:56.792348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:56.792290    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:24:57.041014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:57.040976    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5gjsv"]
Sep 10 17:24:57.044458 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:24:57.044434    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced14a44_5fd5_4179_af31_eef5392a3e7c.slice/crio-6d39982ef2531b7d9e54d9dd0c51f72abe7053afb278445238d0af476093ed73 WatchSource:0}: Error finding container 6d39982ef2531b7d9e54d9dd0c51f72abe7053afb278445238d0af476093ed73: Status 404 returned error can't find the container with id 6d39982ef2531b7d9e54d9dd0c51f72abe7053afb278445238d0af476093ed73
Sep 10 17:24:57.061455 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:57.061431    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njlj5"]
Sep 10 17:24:57.064081 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:24:57.064057    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbade337_a7dc_4cab_9479_2303873983b3.slice/crio-8d403a8394bfb46b3a71595d969c66c8f77d36bf0bfa5c0d4138752b8b7c5a8c WatchSource:0}: Error finding container 8d403a8394bfb46b3a71595d969c66c8f77d36bf0bfa5c0d4138752b8b7c5a8c: Status 404 returned error can't find the container with id 8d403a8394bfb46b3a71595d969c66c8f77d36bf0bfa5c0d4138752b8b7c5a8c
Sep 10 17:24:57.101221 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:57.101190    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njlj5" event={"ID":"cbade337-a7dc-4cab-9479-2303873983b3","Type":"ContainerStarted","Data":"8d403a8394bfb46b3a71595d969c66c8f77d36bf0bfa5c0d4138752b8b7c5a8c"}
Sep 10 17:24:57.102328 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:57.102300    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5gjsv" event={"ID":"ced14a44-5fd5-4179-af31-eef5392a3e7c","Type":"ContainerStarted","Data":"6d39982ef2531b7d9e54d9dd0c51f72abe7053afb278445238d0af476093ed73"}
Sep 10 17:24:57.546605 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:24:57.546566    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:24:57.547094 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:57.546712    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:24:57.547094 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:24:57.546790    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls podName:412fe3b2-c273-4b82-b8a9-cf0559396157 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:29.546767981 +0000 UTC m=+98.256362228 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls") pod "dns-default-8lqdv" (UID: "412fe3b2-c273-4b82-b8a9-cf0559396157") : secret "dns-default-metrics-tls" not found
Sep 10 17:25:02.117279 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:02.117224    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5gjsv" event={"ID":"ced14a44-5fd5-4179-af31-eef5392a3e7c","Type":"ContainerStarted","Data":"4b68fdedcd8cb161d8777423430d7ebd9c044eaf742763003bff18a2edbca491"}
Sep 10 17:25:02.118656 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:02.118632    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njlj5" event={"ID":"cbade337-a7dc-4cab-9479-2303873983b3","Type":"ContainerStarted","Data":"9f0ed2d497bbee64297f188feb36badef2a347667a6b952d671fd5c9ef5acff5"}
Sep 10 17:25:02.118790 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:02.118772    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:25:02.133742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:02.133687    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5gjsv" podStartSLOduration=65.436065878 podStartE2EDuration="1m10.133670937s" totalImagesPullingTime="4.697605059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:02.133321308 +0000 UTC m=+70.842915575" watchObservedRunningTime="2026-09-10 17:25:02.133670937 +0000 UTC m=+70.843265205"
Sep 10 17:25:02.149621 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:02.149564    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njlj5" podStartSLOduration=65.483226225 podStartE2EDuration="1m10.149547818s" totalImagesPullingTime="4.666321593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:02.149367057 +0000 UTC m=+70.858961322" watchObservedRunningTime="2026-09-10 17:25:02.149547818 +0000 UTC m=+70.859142077"
Sep 10 17:25:27.696666 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.696545    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-d8j9l"]
Sep 10 17:25:27.699342 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.699326    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-d8j9l"
Sep 10 17:25:27.701947 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.701930    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cl22n\""
Sep 10 17:25:27.710475 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.710451    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6568874df4-c5t25"]
Sep 10 17:25:27.712077 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.712063    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.715229 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.715210    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 17:25:27.715344 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.715210    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kwhpq\""
Sep 10 17:25:27.715563 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.715540    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 17:25:27.715768 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.715753    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 17:25:27.717627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.717608    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-d8j9l"]
Sep 10 17:25:27.724047 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.724028    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 17:25:27.738131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.738111    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6568874df4-c5t25"]
Sep 10 17:25:27.749151 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749131    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-bound-sa-token\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.749265 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749157    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg82m\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-kube-api-access-qg82m\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.749265 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749175    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rp6s\" (UniqueName: \"kubernetes.io/projected/1e5c87de-e566-44c7-896d-c323a1c4483e-kube-api-access-6rp6s\") pod \"network-check-source-685456655f-d8j9l\" (UID: \"1e5c87de-e566-44c7-896d-c323a1c4483e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-d8j9l"
Sep 10 17:25:27.749265 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-image-registry-private-configuration\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.749372 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749308    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-installation-pull-secrets\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.749372 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749336    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.749372 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749364    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-certificates\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.749466 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749413    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-ca-trust-extracted\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.749466 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.749433    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-trusted-ca\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.795057 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.795022    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2"]
Sep 10 17:25:27.796865 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.796850    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2"
Sep 10 17:25:27.799575 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.799556    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 17:25:27.799689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.799668    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:25:27.799731 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.799668    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x9pwr\""
Sep 10 17:25:27.808825 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.808798    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2"]
Sep 10 17:25:27.850044 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850044 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850049    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpfz\" (UniqueName: \"kubernetes.io/projected/6946ebd9-258c-40f2-ac75-fe7e99aafb1b-kube-api-access-ltpfz\") pod \"volume-data-source-validator-695c895bd6-bxzh2\" (UID: \"6946ebd9-258c-40f2-ac75-fe7e99aafb1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2"
Sep 10 17:25:27.850326 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850074    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-certificates\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850326 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850100    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-ca-trust-extracted\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850326 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850118    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-trusted-ca\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850326 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850147    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-bound-sa-token\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850326 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850166    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg82m\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-kube-api-access-qg82m\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850326 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:27.850175    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:25:27.850326 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:27.850202    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6568874df4-c5t25: secret "image-registry-tls" not found
Sep 10 17:25:27.850326 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:27.850285    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls podName:d88fc6ff-8583-4b84-9dcf-70b20c7879e8 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:28.350268864 +0000 UTC m=+97.059863112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls") pod "image-registry-6568874df4-c5t25" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8") : secret "image-registry-tls" not found
Sep 10 17:25:27.850750 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850188    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rp6s\" (UniqueName: \"kubernetes.io/projected/1e5c87de-e566-44c7-896d-c323a1c4483e-kube-api-access-6rp6s\") pod \"network-check-source-685456655f-d8j9l\" (UID: \"1e5c87de-e566-44c7-896d-c323a1c4483e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-d8j9l"
Sep 10 17:25:27.850750 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850430    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-image-registry-private-configuration\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850750 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850480    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-installation-pull-secrets\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850750 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850588    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-ca-trust-extracted\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.850935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.850760    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-certificates\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.851268 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.851228    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-trusted-ca\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.852896 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.852875    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-image-registry-private-configuration\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.852984 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.852921    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-installation-pull-secrets\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.859187 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.859160    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg82m\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-kube-api-access-qg82m\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.859358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.859343    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-bound-sa-token\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:27.859592 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.859572    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rp6s\" (UniqueName: \"kubernetes.io/projected/1e5c87de-e566-44c7-896d-c323a1c4483e-kube-api-access-6rp6s\") pod \"network-check-source-685456655f-d8j9l\" (UID: \"1e5c87de-e566-44c7-896d-c323a1c4483e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-d8j9l"
Sep 10 17:25:27.951598 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.951499    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpfz\" (UniqueName: \"kubernetes.io/projected/6946ebd9-258c-40f2-ac75-fe7e99aafb1b-kube-api-access-ltpfz\") pod \"volume-data-source-validator-695c895bd6-bxzh2\" (UID: \"6946ebd9-258c-40f2-ac75-fe7e99aafb1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2"
Sep 10 17:25:27.960645 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:27.960623    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpfz\" (UniqueName: \"kubernetes.io/projected/6946ebd9-258c-40f2-ac75-fe7e99aafb1b-kube-api-access-ltpfz\") pod \"volume-data-source-validator-695c895bd6-bxzh2\" (UID: \"6946ebd9-258c-40f2-ac75-fe7e99aafb1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2"
Sep 10 17:25:28.007615 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:28.007577    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-d8j9l"
Sep 10 17:25:28.105809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:28.105768    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2"
Sep 10 17:25:28.275336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:28.275309    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-d8j9l"]
Sep 10 17:25:28.278155 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:25:28.278123    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5c87de_e566_44c7_896d_c323a1c4483e.slice/crio-76c0967957d28f9a229f9a037d2651f50871a5bd923c95e97fc181f6b9ed0543 WatchSource:0}: Error finding container 76c0967957d28f9a229f9a037d2651f50871a5bd923c95e97fc181f6b9ed0543: Status 404 returned error can't find the container with id 76c0967957d28f9a229f9a037d2651f50871a5bd923c95e97fc181f6b9ed0543
Sep 10 17:25:28.355403 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:28.355373    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:28.355582 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:28.355513    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:25:28.355582 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:28.355532    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6568874df4-c5t25: secret "image-registry-tls" not found
Sep 10 17:25:28.355696 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:28.355613    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls podName:d88fc6ff-8583-4b84-9dcf-70b20c7879e8 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:29.355574727 +0000 UTC m=+98.065168994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls") pod "image-registry-6568874df4-c5t25" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8") : secret "image-registry-tls" not found
Sep 10 17:25:28.369703 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:28.369674    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2"]
Sep 10 17:25:28.373169 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:25:28.373143    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6946ebd9_258c_40f2_ac75_fe7e99aafb1b.slice/crio-01d9ff66ad7c872efc73e84cefc6765fb2a38e638569e58502390ba178a31d8d WatchSource:0}: Error finding container 01d9ff66ad7c872efc73e84cefc6765fb2a38e638569e58502390ba178a31d8d: Status 404 returned error can't find the container with id 01d9ff66ad7c872efc73e84cefc6765fb2a38e638569e58502390ba178a31d8d
Sep 10 17:25:29.185577 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:29.185537    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2" event={"ID":"6946ebd9-258c-40f2-ac75-fe7e99aafb1b","Type":"ContainerStarted","Data":"01d9ff66ad7c872efc73e84cefc6765fb2a38e638569e58502390ba178a31d8d"}
Sep 10 17:25:29.187197 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:29.187157    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-d8j9l" event={"ID":"1e5c87de-e566-44c7-896d-c323a1c4483e","Type":"ContainerStarted","Data":"6b374c7be2fec9e7241bbe0ea957695f9ff986cf976739e07054c824ac030549"}
Sep 10 17:25:29.187334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:29.187206    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-d8j9l" event={"ID":"1e5c87de-e566-44c7-896d-c323a1c4483e","Type":"ContainerStarted","Data":"76c0967957d28f9a229f9a037d2651f50871a5bd923c95e97fc181f6b9ed0543"}
Sep 10 17:25:29.204891 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:29.204829    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-d8j9l" podStartSLOduration=2.2048083099999998 podStartE2EDuration="2.20480831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:29.203454987 +0000 UTC m=+97.913049252" watchObservedRunningTime="2026-09-10 17:25:29.20480831 +0000 UTC m=+97.914402573"
Sep 10 17:25:29.362247 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:29.362192    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:29.362413 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:29.362372    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:25:29.362503 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:29.362490    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6568874df4-c5t25: secret "image-registry-tls" not found
Sep 10 17:25:29.362672 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:29.362643    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls podName:d88fc6ff-8583-4b84-9dcf-70b20c7879e8 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:31.362594138 +0000 UTC m=+100.072188394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls") pod "image-registry-6568874df4-c5t25" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8") : secret "image-registry-tls" not found
Sep 10 17:25:29.563885 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:29.563842    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:25:29.564067 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:29.564015    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:25:29.564131 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:29.564090    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls podName:412fe3b2-c273-4b82-b8a9-cf0559396157 nodeName:}" failed. No retries permitted until 2026-09-10 17:26:33.564067933 +0000 UTC m=+162.273662176 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls") pod "dns-default-8lqdv" (UID: "412fe3b2-c273-4b82-b8a9-cf0559396157") : secret "dns-default-metrics-tls" not found
Sep 10 17:25:30.192313 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:30.192280    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2" event={"ID":"6946ebd9-258c-40f2-ac75-fe7e99aafb1b","Type":"ContainerStarted","Data":"ac68d538f7a394bbc1962ca79c539125668de948d7b7b03706d439a623ab2ed3"}
Sep 10 17:25:30.209644 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:30.209595    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bxzh2" podStartSLOduration=1.9667080110000001 podStartE2EDuration="3.20957939s" totalImagesPullingTime="1.242871379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:30.209054872 +0000 UTC m=+98.918649137" watchObservedRunningTime="2026-09-10 17:25:30.20957939 +0000 UTC m=+98.919173669"
Sep 10 17:25:31.377482 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.377445    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:31.377877 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:31.377593    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:25:31.377877 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:31.377611    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6568874df4-c5t25: secret "image-registry-tls" not found
Sep 10 17:25:31.377877 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:31.377665    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls podName:d88fc6ff-8583-4b84-9dcf-70b20c7879e8 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:35.377650241 +0000 UTC m=+104.087244485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls") pod "image-registry-6568874df4-c5t25" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8") : secret "image-registry-tls" not found
Sep 10 17:25:31.909165 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.909136    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mkd2r"]
Sep 10 17:25:31.929940 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.929911    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mkd2r"]
Sep 10 17:25:31.930073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.930035    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:31.933227 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.933195    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 17:25:31.934569 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.934546    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-82xn6\""
Sep 10 17:25:31.934689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.934620    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 17:25:31.934689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.934662    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 17:25:31.935611 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.935592    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 17:25:31.981163 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.981135    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05fb386-89c1-46e6-84c0-e494741965e1-data-volume\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:31.981298 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.981216    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:31.981298 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.981256    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:31.981429 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.981317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b05fb386-89c1-46e6-84c0-e494741965e1-crio-socket\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:31.981429 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:31.981369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxcj\" (UniqueName: \"kubernetes.io/projected/b05fb386-89c1-46e6-84c0-e494741965e1-kube-api-access-vvxcj\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.082606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.082577    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.082606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.082608    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.082821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.082628    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b05fb386-89c1-46e6-84c0-e494741965e1-crio-socket\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.082821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.082659    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxcj\" (UniqueName: \"kubernetes.io/projected/b05fb386-89c1-46e6-84c0-e494741965e1-kube-api-access-vvxcj\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.082821 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:32.082723    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:32.082821 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:32.082756    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:32.082821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.082778    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05fb386-89c1-46e6-84c0-e494741965e1-data-volume\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.082821 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:32.082794    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:32.582778799 +0000 UTC m=+101.292373046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:32.082821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.082756    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b05fb386-89c1-46e6-84c0-e494741965e1-crio-socket\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.082821 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:32.082826    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:32.582809705 +0000 UTC m=+101.292403969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:32.083159 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.082989    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05fb386-89c1-46e6-84c0-e494741965e1-data-volume\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.098100 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.098075    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxcj\" (UniqueName: \"kubernetes.io/projected/b05fb386-89c1-46e6-84c0-e494741965e1-kube-api-access-vvxcj\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.167575 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.167501    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf"]
Sep 10 17:25:32.183776 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.183742    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf"]
Sep 10 17:25:32.183776 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.183765    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf"
Sep 10 17:25:32.186596 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.186572    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 17:25:32.186701 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.186577    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 17:25:32.186701 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.186613    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6nz2h\""
Sep 10 17:25:32.284697 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.284661    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg5vq\" (UniqueName: \"kubernetes.io/projected/018f80dc-d002-4a7b-8d8b-c8335aecfa20-kube-api-access-bg5vq\") pod \"migrator-fdbb8fdff-9l9bf\" (UID: \"018f80dc-d002-4a7b-8d8b-c8335aecfa20\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf"
Sep 10 17:25:32.385273 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.385218    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg5vq\" (UniqueName: \"kubernetes.io/projected/018f80dc-d002-4a7b-8d8b-c8335aecfa20-kube-api-access-bg5vq\") pod \"migrator-fdbb8fdff-9l9bf\" (UID: \"018f80dc-d002-4a7b-8d8b-c8335aecfa20\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf"
Sep 10 17:25:32.393980 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.393954    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg5vq\" (UniqueName: \"kubernetes.io/projected/018f80dc-d002-4a7b-8d8b-c8335aecfa20-kube-api-access-bg5vq\") pod \"migrator-fdbb8fdff-9l9bf\" (UID: \"018f80dc-d002-4a7b-8d8b-c8335aecfa20\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf"
Sep 10 17:25:32.492698 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.492608    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf"
Sep 10 17:25:32.586816 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.586788    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.586952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.586820    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:32.586952 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:32.586912    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:32.587018 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:32.586974    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:33.586958079 +0000 UTC m=+102.296552350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:32.587018 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:32.586921    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:32.587091 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:32.587035    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:33.587023431 +0000 UTC m=+102.296617675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:32.792031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:32.791987    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf"]
Sep 10 17:25:32.794351 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:25:32.794326    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018f80dc_d002_4a7b_8d8b_c8335aecfa20.slice/crio-29b79428c8db772de3d41ba895ebc7efdd0c7e2ea24bfd7055e0799e31f92094 WatchSource:0}: Error finding container 29b79428c8db772de3d41ba895ebc7efdd0c7e2ea24bfd7055e0799e31f92094: Status 404 returned error can't find the container with id 29b79428c8db772de3d41ba895ebc7efdd0c7e2ea24bfd7055e0799e31f92094
Sep 10 17:25:33.123132 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:33.123045    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njlj5"
Sep 10 17:25:33.201585 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:33.201554    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf" event={"ID":"018f80dc-d002-4a7b-8d8b-c8335aecfa20","Type":"ContainerStarted","Data":"29b79428c8db772de3d41ba895ebc7efdd0c7e2ea24bfd7055e0799e31f92094"}
Sep 10 17:25:33.595461 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:33.595426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:33.595461 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:33.595468    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:33.596037 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:33.595576    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:33.596037 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:33.595601    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:33.596037 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:33.595659    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:35.595640218 +0000 UTC m=+104.305234465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:33.596037 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:33.595681    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:35.595669846 +0000 UTC m=+104.305264090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:33.650898 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:33.650865    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2pdpk_f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54/dns-node-resolver/0.log"
Sep 10 17:25:34.206500 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:34.206463    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf" event={"ID":"018f80dc-d002-4a7b-8d8b-c8335aecfa20","Type":"ContainerStarted","Data":"ff90ae8ecf0d36acac6302bb582b19067908700c11723bbbae35a3ee9f450776"}
Sep 10 17:25:34.206691 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:34.206517    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf" event={"ID":"018f80dc-d002-4a7b-8d8b-c8335aecfa20","Type":"ContainerStarted","Data":"70d9c60d98189505f85ef99d5ba7ba76c232424471a4ee149621bb5dbf50c288"}
Sep 10 17:25:34.223381 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:34.223338    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-9l9bf" podStartSLOduration=0.995370033 podStartE2EDuration="2.223323s" totalImagesPullingTime="1.227952967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:34.22270259 +0000 UTC m=+102.932296857" watchObservedRunningTime="2026-09-10 17:25:34.223323 +0000 UTC m=+102.932917267"
Sep 10 17:25:34.651358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:34.651331    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rtwg_63e68c07-a8f9-4182-a40c-a1ef1470e7cc/node-ca/0.log"
Sep 10 17:25:35.408462 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:35.408423    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:35.408628 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:35.408534    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:25:35.408628 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:35.408545    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6568874df4-c5t25: secret "image-registry-tls" not found
Sep 10 17:25:35.408628 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:35.408594    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls podName:d88fc6ff-8583-4b84-9dcf-70b20c7879e8 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:43.408580636 +0000 UTC m=+112.118174879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls") pod "image-registry-6568874df4-c5t25" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8") : secret "image-registry-tls" not found
Sep 10 17:25:35.609957 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:35.609921    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:35.609957 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:35.609958    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:35.610161 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:35.610044    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:35.610161 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:35.610070    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:35.610161 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:35.610116    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:39.610094856 +0000 UTC m=+108.319689116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:35.610161 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:35.610130    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:39.610124447 +0000 UTC m=+108.319718691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:35.652414 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:35.652386    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhlnf_945e40c2-0163-4303-98ca-1519c942c638/serve-healthcheck-canary/0.log"
Sep 10 17:25:39.640342 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:39.640294    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:39.640342 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:39.640344    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:39.640776 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:39.640421    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:25:39.640776 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:39.640477    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:39.640776 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:39.640498    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:47.640482421 +0000 UTC m=+116.350076666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : configmap "kube-rbac-proxy" not found
Sep 10 17:25:39.640776 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:25:39.640529    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls podName:b05fb386-89c1-46e6-84c0-e494741965e1 nodeName:}" failed. No retries permitted until 2026-09-10 17:25:47.640517755 +0000 UTC m=+116.350111999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkd2r" (UID: "b05fb386-89c1-46e6-84c0-e494741965e1") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:25:43.466606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:43.466559    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:43.469071 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:43.469045    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"image-registry-6568874df4-c5t25\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") " pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:43.620734 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:43.620694    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:43.898822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:43.898790    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6568874df4-c5t25"]
Sep 10 17:25:43.901999 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:25:43.901969    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88fc6ff_8583_4b84_9dcf_70b20c7879e8.slice/crio-5bc303a0871d8fb7dd0e72b572d058078d496990ce0c1476eb3608444a9c3f75 WatchSource:0}: Error finding container 5bc303a0871d8fb7dd0e72b572d058078d496990ce0c1476eb3608444a9c3f75: Status 404 returned error can't find the container with id 5bc303a0871d8fb7dd0e72b572d058078d496990ce0c1476eb3608444a9c3f75
Sep 10 17:25:44.233755 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:44.233671    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6568874df4-c5t25" event={"ID":"d88fc6ff-8583-4b84-9dcf-70b20c7879e8","Type":"ContainerStarted","Data":"3170c7d9a3da611e37c51d716baf6ad8c4eba1af349869ec9b8386ef6a9244a5"}
Sep 10 17:25:44.233755 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:44.233716    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6568874df4-c5t25" event={"ID":"d88fc6ff-8583-4b84-9dcf-70b20c7879e8","Type":"ContainerStarted","Data":"5bc303a0871d8fb7dd0e72b572d058078d496990ce0c1476eb3608444a9c3f75"}
Sep 10 17:25:44.233927 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:44.233800    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:25:44.256848 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:44.256806    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6568874df4-c5t25" podStartSLOduration=17.25679163 podStartE2EDuration="17.25679163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:44.255772902 +0000 UTC m=+112.965367169" watchObservedRunningTime="2026-09-10 17:25:44.25679163 +0000 UTC m=+112.966385895"
Sep 10 17:25:47.702214 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:47.702176    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:47.702596 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:47.702219    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:47.702851 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:47.702830    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b05fb386-89c1-46e6-84c0-e494741965e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:47.704517 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:47.704500    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b05fb386-89c1-46e6-84c0-e494741965e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkd2r\" (UID: \"b05fb386-89c1-46e6-84c0-e494741965e1\") " pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:47.841414 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:47.841363    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mkd2r"
Sep 10 17:25:48.116423 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:48.116388    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mkd2r"]
Sep 10 17:25:48.119424 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:25:48.119396    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05fb386_89c1_46e6_84c0_e494741965e1.slice/crio-ae03f074873e381948f495d730cfc81411099dd5c0eefc9a8b1777c2649cd083 WatchSource:0}: Error finding container ae03f074873e381948f495d730cfc81411099dd5c0eefc9a8b1777c2649cd083: Status 404 returned error can't find the container with id ae03f074873e381948f495d730cfc81411099dd5c0eefc9a8b1777c2649cd083
Sep 10 17:25:48.248760 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:48.248725    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mkd2r" event={"ID":"b05fb386-89c1-46e6-84c0-e494741965e1","Type":"ContainerStarted","Data":"a2a89f3dbed839c7f7f2464431257babaf58c70699cc6dbbee1267d37992ea8c"}
Sep 10 17:25:48.248900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:48.248791    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mkd2r" event={"ID":"b05fb386-89c1-46e6-84c0-e494741965e1","Type":"ContainerStarted","Data":"ae03f074873e381948f495d730cfc81411099dd5c0eefc9a8b1777c2649cd083"}
Sep 10 17:25:49.253632 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:49.253547    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mkd2r" event={"ID":"b05fb386-89c1-46e6-84c0-e494741965e1","Type":"ContainerStarted","Data":"06a07b64ecb57c4c7fb70c665aa8979623e355df5202fd0490a2538ddb09ca76"}
Sep 10 17:25:51.262086 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:51.262050    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mkd2r" event={"ID":"b05fb386-89c1-46e6-84c0-e494741965e1","Type":"ContainerStarted","Data":"2f45eea68220276d43b7a9b9374ea092527a315ce0afc84b4fad77f816f22d9b"}
Sep 10 17:25:51.281905 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:51.281858    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mkd2r" podStartSLOduration=18.124050615 podStartE2EDuration="20.281843817s" totalImagesPullingTime="2.157793202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:25:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:51.281336683 +0000 UTC m=+119.990930948" watchObservedRunningTime="2026-09-10 17:25:51.281843817 +0000 UTC m=+119.991438083"
Sep 10 17:25:57.237208 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.237171    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"]
Sep 10 17:25:57.240508 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.240492    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.242986 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.242961    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 17:25:57.243099 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.242965    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 17:25:57.243308 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.243289    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6568874df4-c5t25"]
Sep 10 17:25:57.244081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.244061    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 17:25:57.244172 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.244061    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 17:25:57.244381 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.244366    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 17:25:57.247493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.247269    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 17:25:57.247916 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.247895    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 17:25:57.260001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.259976    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"]
Sep 10 17:25:57.375500 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.375460    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.375500 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.375497    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lq9\" (UniqueName: \"kubernetes.io/projected/8d6bfb40-c397-4fa9-aa16-042a84554a69-kube-api-access-k4lq9\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.375723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.375519    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-hub\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.375723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.375579    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-ca\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.375723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.375609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6bfb40-c397-4fa9-aa16-042a84554a69-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.375723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.375627    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.476788 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.476758    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-ca\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.476900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.476794    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6bfb40-c397-4fa9-aa16-042a84554a69-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.476900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.476815    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.476900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.476858    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.476900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.476874    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lq9\" (UniqueName: \"kubernetes.io/projected/8d6bfb40-c397-4fa9-aa16-042a84554a69-kube-api-access-k4lq9\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.476900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.476892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-hub\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.477639 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.477586    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6bfb40-c397-4fa9-aa16-042a84554a69-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.479213 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.479186    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-ca\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.479465 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.479447    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.479530 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.479498    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.479582 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.479530    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d6bfb40-c397-4fa9-aa16-042a84554a69-hub\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.489544 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.489491    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lq9\" (UniqueName: \"kubernetes.io/projected/8d6bfb40-c397-4fa9-aa16-042a84554a69-kube-api-access-k4lq9\") pod \"cluster-proxy-proxy-agent-d79cc79cf-6hm7s\" (UID: \"8d6bfb40-c397-4fa9-aa16-042a84554a69\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.558352 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.558316    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"
Sep 10 17:25:57.840725 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:57.840694    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s"]
Sep 10 17:25:57.843743 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:25:57.843714    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d6bfb40_c397_4fa9_aa16_042a84554a69.slice/crio-ada003a7fcaa5b23b8f804225351f447c4784b44afa4d48a9993089eb70e608f WatchSource:0}: Error finding container ada003a7fcaa5b23b8f804225351f447c4784b44afa4d48a9993089eb70e608f: Status 404 returned error can't find the container with id ada003a7fcaa5b23b8f804225351f447c4784b44afa4d48a9993089eb70e608f
Sep 10 17:25:58.285959 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:25:58.285926    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s" event={"ID":"8d6bfb40-c397-4fa9-aa16-042a84554a69","Type":"ContainerStarted","Data":"ada003a7fcaa5b23b8f804225351f447c4784b44afa4d48a9993089eb70e608f"}
Sep 10 17:26:00.604742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:00.604703    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:26:00.607344 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:00.607315    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ea78e61-4de5-43fa-81a2-8ffff13e1d3a-metrics-certs\") pod \"network-metrics-daemon-gl4v7\" (UID: \"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a\") " pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:26:00.678274 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:00.678222    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qjn9k\""
Sep 10 17:26:00.686300 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:00.686267    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gl4v7"
Sep 10 17:26:01.570819 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:01.570695    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gl4v7"]
Sep 10 17:26:01.573250 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:26:01.573199    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea78e61_4de5_43fa_81a2_8ffff13e1d3a.slice/crio-0b52aac822348a8fdd742d0b75f3932c3f8fe0aab6346049c7263751ac18ac32 WatchSource:0}: Error finding container 0b52aac822348a8fdd742d0b75f3932c3f8fe0aab6346049c7263751ac18ac32: Status 404 returned error can't find the container with id 0b52aac822348a8fdd742d0b75f3932c3f8fe0aab6346049c7263751ac18ac32
Sep 10 17:26:02.302553 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:02.302510    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s" event={"ID":"8d6bfb40-c397-4fa9-aa16-042a84554a69","Type":"ContainerStarted","Data":"9bee834f4138727f477442cbb289062d417309c93a0ca3e53ac377bf1149e8c8"}
Sep 10 17:26:02.303014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:02.302567    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s" event={"ID":"8d6bfb40-c397-4fa9-aa16-042a84554a69","Type":"ContainerStarted","Data":"2422678df6ec781bbac60701231c01c447f65219e7221e98434ba1dacf0604fc"}
Sep 10 17:26:02.303014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:02.302589    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s" event={"ID":"8d6bfb40-c397-4fa9-aa16-042a84554a69","Type":"ContainerStarted","Data":"e9f976130dcf3ccdf79325200caccea6ee906a7fbc44f3ef7c310a563b4eed9f"}
Sep 10 17:26:02.303902 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:02.303864    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gl4v7" event={"ID":"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a","Type":"ContainerStarted","Data":"0b52aac822348a8fdd742d0b75f3932c3f8fe0aab6346049c7263751ac18ac32"}
Sep 10 17:26:02.323525 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:02.323457    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d79cc79cf-6hm7s" podStartSLOduration=1.792625849 podStartE2EDuration="5.323434501s" totalImagesPullingTime="3.530808652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:02.321904319 +0000 UTC m=+131.031498585" watchObservedRunningTime="2026-09-10 17:26:02.323434501 +0000 UTC m=+131.033028769"
Sep 10 17:26:03.308856 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:03.308813    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gl4v7" event={"ID":"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a","Type":"ContainerStarted","Data":"8761b2f130c28da89763962f6655413a4eea95afb9db334038541181d7f35bff"}
Sep 10 17:26:03.309326 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:03.308873    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gl4v7" event={"ID":"3ea78e61-4de5-43fa-81a2-8ffff13e1d3a","Type":"ContainerStarted","Data":"d96bdb7b9658c1eb7cc773a5f521213928cd4684470f674ea5fc2dbd4e78271e"}
Sep 10 17:26:03.326107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:03.326043    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gl4v7" podStartSLOduration=130.484186777 podStartE2EDuration="2m11.326021388s" totalImagesPullingTime="841.834611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:03.325376395 +0000 UTC m=+132.034970661" watchObservedRunningTime="2026-09-10 17:26:03.326021388 +0000 UTC m=+132.035615655"
Sep 10 17:26:07.249446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:07.249406    2568 patch_prober.go:28] interesting pod/image-registry-6568874df4-c5t25 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 17:26:07.249853 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:07.249493    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6568874df4-c5t25" podUID="d88fc6ff-8583-4b84-9dcf-70b20c7879e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:26:11.564498 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.564461    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"]
Sep 10 17:26:11.567290 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.567270    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.570339 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.570316    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 17:26:11.570957 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.570940    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 17:26:11.571040 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.570944    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 17:26:11.571662 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.571645    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gj265\""
Sep 10 17:26:11.572187 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.572169    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 17:26:11.572187 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.572178    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 17:26:11.572406 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.572178    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 17:26:11.591026 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.591005    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"]
Sep 10 17:26:11.597355 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.597336    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pmfzv"]
Sep 10 17:26:11.599427 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.599413    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.602050 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.602032    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 17:26:11.602149 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.602056    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 17:26:11.602306 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.602293    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 17:26:11.603732 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.603717    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v58mw\""
Sep 10 17:26:11.687508 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-wtmp\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.687508 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687513    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.687746 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687541    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.687746 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687630    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-textfile\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.687746 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmc9\" (UniqueName: \"kubernetes.io/projected/f1adb054-3538-436e-af61-d94dddb380f4-kube-api-access-ssmc9\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.687746 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687706    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44hm\" (UniqueName: \"kubernetes.io/projected/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-api-access-d44hm\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.687746 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687736    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-root\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.687923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687773    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.687923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687796    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1adb054-3538-436e-af61-d94dddb380f4-metrics-client-ca\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.687923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.687923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687867    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.687923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687899    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.687923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687916    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-sys\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.688171 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.687950    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-tls\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.688171 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.688020    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788463 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788429    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-textfile\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788463 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788464    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmc9\" (UniqueName: \"kubernetes.io/projected/f1adb054-3538-436e-af61-d94dddb380f4-kube-api-access-ssmc9\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788671 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44hm\" (UniqueName: \"kubernetes.io/projected/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-api-access-d44hm\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.788716 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788680    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-root\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788756 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788733    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.788807 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788762    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1adb054-3538-436e-af61-d94dddb380f4-metrics-client-ca\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788807 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788774    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-root\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788888 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788819    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.788888 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788850    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788888 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788859    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-textfile\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788913    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.788998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788943    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-sys\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.788998 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:26:11.788949    2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 10 17:26:11.788998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788970    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-tls\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.789191 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.788999    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.789191 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:26:11.789029    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-tls podName:c1a2785a-f374-4ccd-8c7b-249bc472a1c7 nodeName:}" failed. No retries permitted until 2026-09-10 17:26:12.289008154 +0000 UTC m=+140.998602419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-kf66d" (UID: "c1a2785a-f374-4ccd-8c7b-249bc472a1c7") : secret "kube-state-metrics-tls" not found
Sep 10 17:26:11.789191 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789074    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-wtmp\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.789191 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789100    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-sys\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.789191 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789112    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.789191 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789108    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.789191 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789163    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.789581 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:26:11.789215    2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 17:26:11.789581 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:26:11.789312    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-tls podName:f1adb054-3538-436e-af61-d94dddb380f4 nodeName:}" failed. No retries permitted until 2026-09-10 17:26:12.289295395 +0000 UTC m=+140.998889661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-tls") pod "node-exporter-pmfzv" (UID: "f1adb054-3538-436e-af61-d94dddb380f4") : secret "node-exporter-tls" not found
Sep 10 17:26:11.789581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789454    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-wtmp\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.789581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789496    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1adb054-3538-436e-af61-d94dddb380f4-metrics-client-ca\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.789581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789524    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.789806 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.789727    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.790036 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.790013    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.791356 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.791335    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.791523 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.791505    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:11.797292 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.797270    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmc9\" (UniqueName: \"kubernetes.io/projected/f1adb054-3538-436e-af61-d94dddb380f4-kube-api-access-ssmc9\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:11.797402 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:11.797386    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44hm\" (UniqueName: \"kubernetes.io/projected/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-api-access-d44hm\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:12.292788 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:12.292751    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:12.292949 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:12.292799    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-tls\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:12.295165 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:12.295132    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1adb054-3538-436e-af61-d94dddb380f4-node-exporter-tls\") pod \"node-exporter-pmfzv\" (UID: \"f1adb054-3538-436e-af61-d94dddb380f4\") " pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:12.295274 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:12.295191    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a2785a-f374-4ccd-8c7b-249bc472a1c7-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-kf66d\" (UID: \"c1a2785a-f374-4ccd-8c7b-249bc472a1c7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:12.476554 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:12.476511    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"
Sep 10 17:26:12.508101 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:12.508071    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pmfzv"
Sep 10 17:26:12.519150 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:26:12.519119    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1adb054_3538_436e_af61_d94dddb380f4.slice/crio-50b03cead308fdb1459914080dab8f1c937f6db72cac55a1969a723ff000382f WatchSource:0}: Error finding container 50b03cead308fdb1459914080dab8f1c937f6db72cac55a1969a723ff000382f: Status 404 returned error can't find the container with id 50b03cead308fdb1459914080dab8f1c937f6db72cac55a1969a723ff000382f
Sep 10 17:26:12.750099 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:12.750073    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-kf66d"]
Sep 10 17:26:12.751942 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:26:12.751913    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a2785a_f374_4ccd_8c7b_249bc472a1c7.slice/crio-e702780fce3536b99f296b0dca6bdcaf0a0ca8e1727d725514bf3b60ecf7af5e WatchSource:0}: Error finding container e702780fce3536b99f296b0dca6bdcaf0a0ca8e1727d725514bf3b60ecf7af5e: Status 404 returned error can't find the container with id e702780fce3536b99f296b0dca6bdcaf0a0ca8e1727d725514bf3b60ecf7af5e
Sep 10 17:26:13.344538 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:13.344494    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmfzv" event={"ID":"f1adb054-3538-436e-af61-d94dddb380f4","Type":"ContainerStarted","Data":"50b03cead308fdb1459914080dab8f1c937f6db72cac55a1969a723ff000382f"}
Sep 10 17:26:13.346066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:13.346022    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d" event={"ID":"c1a2785a-f374-4ccd-8c7b-249bc472a1c7","Type":"ContainerStarted","Data":"e702780fce3536b99f296b0dca6bdcaf0a0ca8e1727d725514bf3b60ecf7af5e"}
Sep 10 17:26:14.351451 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:14.351407    2568 generic.go:356] "Generic (PLEG): container finished" podID="f1adb054-3538-436e-af61-d94dddb380f4" containerID="56393ad53dfb6a15fa8fbb5af7209f07a50c541170300a27c811ca68e4ab0824" exitCode=0
Sep 10 17:26:14.351934 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:14.351462    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmfzv" event={"ID":"f1adb054-3538-436e-af61-d94dddb380f4","Type":"ContainerDied","Data":"56393ad53dfb6a15fa8fbb5af7209f07a50c541170300a27c811ca68e4ab0824"}
Sep 10 17:26:14.354327 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:14.354297    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d" event={"ID":"c1a2785a-f374-4ccd-8c7b-249bc472a1c7","Type":"ContainerStarted","Data":"bc301e41c46797b3a80159602e8f2125612754c551ecff344ee1a370d9209996"}
Sep 10 17:26:14.354440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:14.354364    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d" event={"ID":"c1a2785a-f374-4ccd-8c7b-249bc472a1c7","Type":"ContainerStarted","Data":"13942cc54381a5fa67f29ac8026a626298f3e57b32c9ddcc9f03fa623430dfdf"}
Sep 10 17:26:14.354440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:14.354391    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d" event={"ID":"c1a2785a-f374-4ccd-8c7b-249bc472a1c7","Type":"ContainerStarted","Data":"acc9b5b00d585a960ff8bbbbdc98aab17e95b4014c675db98406b6c173969252"}
Sep 10 17:26:14.390741 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:14.390685    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-kf66d" podStartSLOduration=2.171541092 podStartE2EDuration="3.390662152s" totalImagesPullingTime="1.21912106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:14.39037491 +0000 UTC m=+143.099969181" watchObservedRunningTime="2026-09-10 17:26:14.390662152 +0000 UTC m=+143.100256418"
Sep 10 17:26:15.359867 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:15.359827    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmfzv" event={"ID":"f1adb054-3538-436e-af61-d94dddb380f4","Type":"ContainerStarted","Data":"7bf8a7880ce2ef867cf47b40415d60f6882ac207edc69a59fc134e0957fc5f56"}
Sep 10 17:26:15.360283 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:15.359912    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmfzv" event={"ID":"f1adb054-3538-436e-af61-d94dddb380f4","Type":"ContainerStarted","Data":"a16fc929946461d152b497019bf522d6bbec2d90c4ed1db7aaa7453d1e46c7d5"}
Sep 10 17:26:15.379811 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:15.379751    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pmfzv" podStartSLOduration=3.217071628 podStartE2EDuration="4.379735898s" totalImagesPullingTime="1.092847122s" totalInitContainerRuntime="69.817148ms" isStatefulPod=true podCreationTimestamp="2026-09-10 17:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:15.378397429 +0000 UTC m=+144.087991694" watchObservedRunningTime="2026-09-10 17:26:15.379735898 +0000 UTC m=+144.089330163"
Sep 10 17:26:16.347795 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.347762    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7695b86884-rrzsb"]
Sep 10 17:26:16.350590 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.350570    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.354400 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.354370    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 17:26:16.354519 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.354435    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 17:26:16.354777 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.354764    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 17:26:16.355534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.355516    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 17:26:16.355631 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.355552    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7pmmj\""
Sep 10 17:26:16.355631 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.355557    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 17:26:16.355734 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.355658    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 17:26:16.355812 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.355798    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 17:26:16.360364 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.360348    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 17:26:16.362196 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.362177    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7695b86884-rrzsb"]
Sep 10 17:26:16.526947 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.526915    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-oauth-serving-cert\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.527109 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.526970    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-config\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.527109 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.526997    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-oauth-config\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.527109 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.527088    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-service-ca\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.527230 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.527146    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-serving-cert\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.527312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.527291    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-trusted-ca-bundle\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.527366 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.527353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8jg\" (UniqueName: \"kubernetes.io/projected/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-kube-api-access-6j8jg\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.628764 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.628687    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-service-ca\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.628764 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.628720    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-serving-cert\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.628764 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.628741    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-trusted-ca-bundle\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.629040 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.628771    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8jg\" (UniqueName: \"kubernetes.io/projected/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-kube-api-access-6j8jg\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.629040 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.628893    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-oauth-serving-cert\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.629040 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.628950    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-config\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.629040 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.628975    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-oauth-config\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.629556 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.629526    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-service-ca\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.629696 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.629673    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-oauth-serving-cert\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.629745 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.629703    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-config\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.629807 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.629782    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-trusted-ca-bundle\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.631253 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.631200    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-serving-cert\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.631356 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.631267    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-oauth-config\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.637139 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.637118    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8jg\" (UniqueName: \"kubernetes.io/projected/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-kube-api-access-6j8jg\") pod \"console-7695b86884-rrzsb\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") " pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.660307 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.660280    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:16.927967 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:16.927895    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7695b86884-rrzsb"]
Sep 10 17:26:16.930773 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:26:16.930744    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369ed57c_78f9_48a3_9ebc_fb0ea2c28a34.slice/crio-184d868b8fa2c882fb32538365b2d16d43223a5d51de0ed4aa5fd5ec142c8653 WatchSource:0}: Error finding container 184d868b8fa2c882fb32538365b2d16d43223a5d51de0ed4aa5fd5ec142c8653: Status 404 returned error can't find the container with id 184d868b8fa2c882fb32538365b2d16d43223a5d51de0ed4aa5fd5ec142c8653
Sep 10 17:26:17.248155 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:17.248079    2568 patch_prober.go:28] interesting pod/image-registry-6568874df4-c5t25 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 17:26:17.248155 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:17.248131    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6568874df4-c5t25" podUID="d88fc6ff-8583-4b84-9dcf-70b20c7879e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:26:17.370700 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:17.370662    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695b86884-rrzsb" event={"ID":"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34","Type":"ContainerStarted","Data":"184d868b8fa2c882fb32538365b2d16d43223a5d51de0ed4aa5fd5ec142c8653"}
Sep 10 17:26:20.383286 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:20.383252    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695b86884-rrzsb" event={"ID":"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34","Type":"ContainerStarted","Data":"ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1"}
Sep 10 17:26:20.402120 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:20.402045    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7695b86884-rrzsb" podStartSLOduration=1.821561583 podStartE2EDuration="4.402028136s" totalImagesPullingTime="2.580466553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:20.400596309 +0000 UTC m=+149.110190576" watchObservedRunningTime="2026-09-10 17:26:20.402028136 +0000 UTC m=+149.111622402"
Sep 10 17:26:22.263311 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.263262    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6568874df4-c5t25" podUID="d88fc6ff-8583-4b84-9dcf-70b20c7879e8" containerName="registry" containerID="cri-o://3170c7d9a3da611e37c51d716baf6ad8c4eba1af349869ec9b8386ef6a9244a5" gracePeriod=30
Sep 10 17:26:22.391911 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.391880    2568 generic.go:356] "Generic (PLEG): container finished" podID="d88fc6ff-8583-4b84-9dcf-70b20c7879e8" containerID="3170c7d9a3da611e37c51d716baf6ad8c4eba1af349869ec9b8386ef6a9244a5" exitCode=0
Sep 10 17:26:22.392045 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.391940    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6568874df4-c5t25" event={"ID":"d88fc6ff-8583-4b84-9dcf-70b20c7879e8","Type":"ContainerDied","Data":"3170c7d9a3da611e37c51d716baf6ad8c4eba1af349869ec9b8386ef6a9244a5"}
Sep 10 17:26:22.498754 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.498730    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:26:22.681606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.681567    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-bound-sa-token\") pod \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") "
Sep 10 17:26:22.681606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.681609    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-certificates\") pod \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") "
Sep 10 17:26:22.681838 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.681651    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg82m\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-kube-api-access-qg82m\") pod \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") "
Sep 10 17:26:22.681838 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.681674    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") pod \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") "
Sep 10 17:26:22.681838 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.681696    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-installation-pull-secrets\") pod \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") "
Sep 10 17:26:22.681838 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.681729    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-image-registry-private-configuration\") pod \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") "
Sep 10 17:26:22.681838 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.681758    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-ca-trust-extracted\") pod \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") "
Sep 10 17:26:22.682066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.681948    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-trusted-ca\") pod \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\" (UID: \"d88fc6ff-8583-4b84-9dcf-70b20c7879e8\") "
Sep 10 17:26:22.682278 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.682229    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d88fc6ff-8583-4b84-9dcf-70b20c7879e8" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:22.682634 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.682606    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d88fc6ff-8583-4b84-9dcf-70b20c7879e8" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:22.684150 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.684119    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-kube-api-access-qg82m" (OuterVolumeSpecName: "kube-api-access-qg82m") pod "d88fc6ff-8583-4b84-9dcf-70b20c7879e8" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8"). InnerVolumeSpecName "kube-api-access-qg82m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:26:22.684270 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.684195    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d88fc6ff-8583-4b84-9dcf-70b20c7879e8" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:26:22.684321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.684291    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d88fc6ff-8583-4b84-9dcf-70b20c7879e8" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:26:22.684398 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.684374    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d88fc6ff-8583-4b84-9dcf-70b20c7879e8" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:26:22.684448 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.684430    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d88fc6ff-8583-4b84-9dcf-70b20c7879e8" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:26:22.690328 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.690306    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d88fc6ff-8583-4b84-9dcf-70b20c7879e8" (UID: "d88fc6ff-8583-4b84-9dcf-70b20c7879e8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:26:22.783298 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.783220    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg82m\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-kube-api-access-qg82m\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:26:22.783298 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.783291    2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:26:22.783298 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.783307    2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-installation-pull-secrets\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:26:22.783507 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.783320    2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-image-registry-private-configuration\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:26:22.783507 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.783333    2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-ca-trust-extracted\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:26:22.783507 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.783344    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-trusted-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:26:22.783507 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.783357    2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-bound-sa-token\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:26:22.783507 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:22.783370    2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d88fc6ff-8583-4b84-9dcf-70b20c7879e8-registry-certificates\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:26:23.396990 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:23.396958    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6568874df4-c5t25"
Sep 10 17:26:23.397405 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:23.396956    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6568874df4-c5t25" event={"ID":"d88fc6ff-8583-4b84-9dcf-70b20c7879e8","Type":"ContainerDied","Data":"5bc303a0871d8fb7dd0e72b572d058078d496990ce0c1476eb3608444a9c3f75"}
Sep 10 17:26:23.397405 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:23.397089    2568 scope.go:117] "RemoveContainer" containerID="3170c7d9a3da611e37c51d716baf6ad8c4eba1af349869ec9b8386ef6a9244a5"
Sep 10 17:26:23.420618 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:23.420584    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6568874df4-c5t25"]
Sep 10 17:26:23.425875 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:23.425849    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6568874df4-c5t25"]
Sep 10 17:26:23.762113 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:23.762035    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d88fc6ff-8583-4b84-9dcf-70b20c7879e8" path="/var/lib/kubelet/pods/d88fc6ff-8583-4b84-9dcf-70b20c7879e8/volumes"
Sep 10 17:26:26.660619 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:26.660573    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:26.661012 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:26.660652    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:26:26.661998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:26.661978    2568 patch_prober.go:28] interesting pod/console-7695b86884-rrzsb container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body=
Sep 10 17:26:26.662047 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:26.662025    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7695b86884-rrzsb" podUID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused"
Sep 10 17:26:28.695975 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:26:28.695919    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8lqdv" podUID="412fe3b2-c273-4b82-b8a9-cf0559396157"
Sep 10 17:26:29.419770 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:29.419744    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:26:33.660227 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:33.660184    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:26:33.662648 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:33.662621    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/412fe3b2-c273-4b82-b8a9-cf0559396157-metrics-tls\") pod \"dns-default-8lqdv\" (UID: \"412fe3b2-c273-4b82-b8a9-cf0559396157\") " pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:26:33.923338 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:33.923231    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vn8bp\""
Sep 10 17:26:33.931482 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:33.931462    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:26:34.205649 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:34.205623    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lqdv"]
Sep 10 17:26:34.208048 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:26:34.208020    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412fe3b2_c273_4b82_b8a9_cf0559396157.slice/crio-ae3f4df2a8e5eeb6376c36221f52195940b3fa52ed653a5231bf3b3ca15b8cf3 WatchSource:0}: Error finding container ae3f4df2a8e5eeb6376c36221f52195940b3fa52ed653a5231bf3b3ca15b8cf3: Status 404 returned error can't find the container with id ae3f4df2a8e5eeb6376c36221f52195940b3fa52ed653a5231bf3b3ca15b8cf3
Sep 10 17:26:34.437504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:34.437468    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lqdv" event={"ID":"412fe3b2-c273-4b82-b8a9-cf0559396157","Type":"ContainerStarted","Data":"ae3f4df2a8e5eeb6376c36221f52195940b3fa52ed653a5231bf3b3ca15b8cf3"}
Sep 10 17:26:36.447219 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:36.447177    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lqdv" event={"ID":"412fe3b2-c273-4b82-b8a9-cf0559396157","Type":"ContainerStarted","Data":"8232eed61362b452cfe8bb936dd659c453a20c9def955453a205a29b110c2040"}
Sep 10 17:26:36.447661 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:36.447253    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lqdv" event={"ID":"412fe3b2-c273-4b82-b8a9-cf0559396157","Type":"ContainerStarted","Data":"65448d279661545a0b551b263c1b4999c03d1a089f8cbfd5c7d8eb0f99d4e53a"}
Sep 10 17:26:36.447661 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:36.447296    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:26:36.465230 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:36.465183    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8lqdv" podStartSLOduration=129.867907849 podStartE2EDuration="2m11.46516338s" totalImagesPullingTime="1.597255531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:36.464847851 +0000 UTC m=+165.174442128" watchObservedRunningTime="2026-09-10 17:26:36.46516338 +0000 UTC m=+165.174757647"
Sep 10 17:26:36.661336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:36.661299    2568 patch_prober.go:28] interesting pod/console-7695b86884-rrzsb container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body=
Sep 10 17:26:36.661491 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:36.661350    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7695b86884-rrzsb" podUID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused"
Sep 10 17:26:46.453099 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:46.453022    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8lqdv"
Sep 10 17:26:46.661274 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:46.661214    2568 patch_prober.go:28] interesting pod/console-7695b86884-rrzsb container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body=
Sep 10 17:26:46.661443 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:46.661293    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7695b86884-rrzsb" podUID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused"
Sep 10 17:26:52.592999 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:52.592964    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7695b86884-rrzsb"]
Sep 10 17:26:55.992359 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:55.992328    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-kf66d_c1a2785a-f374-4ccd-8c7b-249bc472a1c7/kube-state-metrics/0.log"
Sep 10 17:26:56.192840 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:56.192810    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-kf66d_c1a2785a-f374-4ccd-8c7b-249bc472a1c7/kube-rbac-proxy-main/0.log"
Sep 10 17:26:56.392314 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:56.392280    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-kf66d_c1a2785a-f374-4ccd-8c7b-249bc472a1c7/kube-rbac-proxy-self/0.log"
Sep 10 17:26:58.192059 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:58.192029    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmfzv_f1adb054-3538-436e-af61-d94dddb380f4/init-textfile/0.log"
Sep 10 17:26:58.392531 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:58.392502    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmfzv_f1adb054-3538-436e-af61-d94dddb380f4/node-exporter/0.log"
Sep 10 17:26:58.591800 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:26:58.591770    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmfzv_f1adb054-3538-436e-af61-d94dddb380f4/kube-rbac-proxy/0.log"
Sep 10 17:27:03.792127 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:03.792098    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7695b86884-rrzsb_369ed57c-78f9-48a3-9ebc-fb0ea2c28a34/console/0.log"
Sep 10 17:27:04.795553 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:04.795524    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lqdv_412fe3b2-c273-4b82-b8a9-cf0559396157/dns/0.log"
Sep 10 17:27:04.995677 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:04.995650    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lqdv_412fe3b2-c273-4b82-b8a9-cf0559396157/kube-rbac-proxy/0.log"
Sep 10 17:27:05.593248 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:05.593217    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2pdpk_f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54/dns-node-resolver/0.log"
Sep 10 17:27:06.592514 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:06.592485    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rtwg_63e68c07-a8f9-4182-a40c-a1ef1470e7cc/node-ca/0.log"
Sep 10 17:27:17.612246 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.612165    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7695b86884-rrzsb" podUID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" containerName="console" containerID="cri-o://ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1" gracePeriod=15
Sep 10 17:27:17.851700 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.851674    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:27:17.988052 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.987962    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-oauth-config\") pod \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") "
Sep 10 17:27:17.988052 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988007    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-serving-cert\") pod \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") "
Sep 10 17:27:17.988052 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988041    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-config\") pod \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") "
Sep 10 17:27:17.988344 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988065    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-oauth-serving-cert\") pod \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") "
Sep 10 17:27:17.988344 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988085    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-service-ca\") pod \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") "
Sep 10 17:27:17.988344 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988105    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-trusted-ca-bundle\") pod \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") "
Sep 10 17:27:17.988493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988337    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j8jg\" (UniqueName: \"kubernetes.io/projected/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-kube-api-access-6j8jg\") pod \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\" (UID: \"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34\") "
Sep 10 17:27:17.988552 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988525    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-config" (OuterVolumeSpecName: "console-config") pod "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" (UID: "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:17.988606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988548    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-service-ca" (OuterVolumeSpecName: "service-ca") pod "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" (UID: "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:17.988654 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988593    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" (UID: "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:17.988654 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988606    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:27:17.988654 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.988600    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" (UID: "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:27:17.990586 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.990554    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" (UID: "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:17.990700 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.990596    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-kube-api-access-6j8jg" (OuterVolumeSpecName: "kube-api-access-6j8jg") pod "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" (UID: "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34"). InnerVolumeSpecName "kube-api-access-6j8jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:17.990700 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:17.990617    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" (UID: "369ed57c-78f9-48a3-9ebc-fb0ea2c28a34"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:18.089321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.089287    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-trusted-ca-bundle\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:27:18.089321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.089314    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j8jg\" (UniqueName: \"kubernetes.io/projected/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-kube-api-access-6j8jg\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:27:18.089321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.089325    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-oauth-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:27:18.089550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.089334    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-console-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:27:18.089550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.089342    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-oauth-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:27:18.089550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.089353    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:27:18.589090 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.589051    2568 generic.go:356] "Generic (PLEG): container finished" podID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" containerID="ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1" exitCode=0
Sep 10 17:27:18.589275 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.589102    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695b86884-rrzsb"
Sep 10 17:27:18.589275 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.589123    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695b86884-rrzsb" event={"ID":"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34","Type":"ContainerDied","Data":"ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1"}
Sep 10 17:27:18.589275 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.589175    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695b86884-rrzsb" event={"ID":"369ed57c-78f9-48a3-9ebc-fb0ea2c28a34","Type":"ContainerDied","Data":"184d868b8fa2c882fb32538365b2d16d43223a5d51de0ed4aa5fd5ec142c8653"}
Sep 10 17:27:18.589275 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.589202    2568 scope.go:117] "RemoveContainer" containerID="ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1"
Sep 10 17:27:18.598211 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.598193    2568 scope.go:117] "RemoveContainer" containerID="ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1"
Sep 10 17:27:18.598529 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:27:18.598505    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1\": container with ID starting with ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1 not found: ID does not exist" containerID="ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1"
Sep 10 17:27:18.598579 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.598538    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1"} err="failed to get container status \"ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1\": rpc error: code = NotFound desc = could not find container \"ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1\": container with ID starting with ef02faa602d7905a68f786828e64b224749039310a2139e65c13232fa100bda1 not found: ID does not exist"
Sep 10 17:27:18.611242 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.611216    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7695b86884-rrzsb"]
Sep 10 17:27:18.615040 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:18.615019    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7695b86884-rrzsb"]
Sep 10 17:27:19.761922 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:19.761890    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" path="/var/lib/kubelet/pods/369ed57c-78f9-48a3-9ebc-fb0ea2c28a34/volumes"
Sep 10 17:27:33.443615 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.443579    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78f7ccd44-5c57j"]
Sep 10 17:27:33.444039 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.443892    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88fc6ff-8583-4b84-9dcf-70b20c7879e8" containerName="registry"
Sep 10 17:27:33.444039 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.443905    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88fc6ff-8583-4b84-9dcf-70b20c7879e8" containerName="registry"
Sep 10 17:27:33.444039 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.443924    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" containerName="console"
Sep 10 17:27:33.444039 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.443929    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" containerName="console"
Sep 10 17:27:33.444039 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.443973    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="369ed57c-78f9-48a3-9ebc-fb0ea2c28a34" containerName="console"
Sep 10 17:27:33.444039 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.443981    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88fc6ff-8583-4b84-9dcf-70b20c7879e8" containerName="registry"
Sep 10 17:27:33.448375 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.448344    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.451162 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.451140    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7pmmj\""
Sep 10 17:27:33.451391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.451375    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 17:27:33.452492 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.452447    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 17:27:33.452492 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.452466    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 17:27:33.452492 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.452501    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 17:27:33.452733 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.452471    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 17:27:33.452733 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.452466    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 17:27:33.452733 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.452646    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 17:27:33.457649 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.457630    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 17:27:33.462535 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.462512    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78f7ccd44-5c57j"]
Sep 10 17:27:33.602392 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.602352    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nxs\" (UniqueName: \"kubernetes.io/projected/3d46d4c8-589e-49c3-a1b8-43be173b3f75-kube-api-access-v7nxs\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.602586 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.602398    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-trusted-ca-bundle\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.602586 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.602470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-oauth-serving-cert\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.602586 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.602554    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-oauth-config\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.602752 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.602619    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-service-ca\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.602752 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.602658    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-config\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.602752 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.602683    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-serving-cert\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.703668 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.703560    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-oauth-serving-cert\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.703668 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.703633    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-oauth-config\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.703668 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.703670    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-service-ca\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.703912 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.703698    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-config\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.703912 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.703838    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-serving-cert\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.703912 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.703885    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nxs\" (UniqueName: \"kubernetes.io/projected/3d46d4c8-589e-49c3-a1b8-43be173b3f75-kube-api-access-v7nxs\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.704059 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.703911    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-trusted-ca-bundle\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.704484 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.704459    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-config\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.704610 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.704459    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-oauth-serving-cert\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.704610 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.704516    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-service-ca\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.704730 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.704667    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-trusted-ca-bundle\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.706210 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.706182    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-oauth-config\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.706504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.706481    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-serving-cert\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.712576 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.712553    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nxs\" (UniqueName: \"kubernetes.io/projected/3d46d4c8-589e-49c3-a1b8-43be173b3f75-kube-api-access-v7nxs\") pod \"console-78f7ccd44-5c57j\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") " pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:33.757634 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:33.757603    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:34.040420 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:34.040382    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78f7ccd44-5c57j"]
Sep 10 17:27:34.045297 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:27:34.045268    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d46d4c8_589e_49c3_a1b8_43be173b3f75.slice/crio-fe49ead90623962040409332cd1dbcc3536ba465e9328d65ea233cd037a179c1 WatchSource:0}: Error finding container fe49ead90623962040409332cd1dbcc3536ba465e9328d65ea233cd037a179c1: Status 404 returned error can't find the container with id fe49ead90623962040409332cd1dbcc3536ba465e9328d65ea233cd037a179c1
Sep 10 17:27:34.649965 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:34.649924    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f7ccd44-5c57j" event={"ID":"3d46d4c8-589e-49c3-a1b8-43be173b3f75","Type":"ContainerStarted","Data":"e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e"}
Sep 10 17:27:34.649965 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:34.649974    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f7ccd44-5c57j" event={"ID":"3d46d4c8-589e-49c3-a1b8-43be173b3f75","Type":"ContainerStarted","Data":"fe49ead90623962040409332cd1dbcc3536ba465e9328d65ea233cd037a179c1"}
Sep 10 17:27:34.668177 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:34.668123    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78f7ccd44-5c57j" podStartSLOduration=1.6681050910000002 podStartE2EDuration="1.668105091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:34.666515974 +0000 UTC m=+223.376110242" watchObservedRunningTime="2026-09-10 17:27:34.668105091 +0000 UTC m=+223.377699356"
Sep 10 17:27:43.761762 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:43.761729    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:43.762137 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:43.761783    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:43.763514 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:43.763492    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:27:44.686553 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:27:44.686518    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:28:42.981300 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:42.981258    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"]
Sep 10 17:28:42.984307 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:42.984288    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"
Sep 10 17:28:42.988571 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:42.988550    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 17:28:42.988933 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:42.988914    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 17:28:42.989664 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:42.989646    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cnb2q\""
Sep 10 17:28:42.996069 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:42.996050    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"]
Sep 10 17:28:43.084323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.084289    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnttw\" (UniqueName: \"kubernetes.io/projected/c07696c4-7e34-4ad7-909d-c094f6dc2b84-kube-api-access-gnttw\") pod \"cert-manager-cainjector-559cdd8c87-5hm25\" (UID: \"c07696c4-7e34-4ad7-909d-c094f6dc2b84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"
Sep 10 17:28:43.084323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.084329    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c07696c4-7e34-4ad7-909d-c094f6dc2b84-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5hm25\" (UID: \"c07696c4-7e34-4ad7-909d-c094f6dc2b84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"
Sep 10 17:28:43.185019 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.184966    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnttw\" (UniqueName: \"kubernetes.io/projected/c07696c4-7e34-4ad7-909d-c094f6dc2b84-kube-api-access-gnttw\") pod \"cert-manager-cainjector-559cdd8c87-5hm25\" (UID: \"c07696c4-7e34-4ad7-909d-c094f6dc2b84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"
Sep 10 17:28:43.185019 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.185024    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c07696c4-7e34-4ad7-909d-c094f6dc2b84-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5hm25\" (UID: \"c07696c4-7e34-4ad7-909d-c094f6dc2b84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"
Sep 10 17:28:43.194012 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.193985    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c07696c4-7e34-4ad7-909d-c094f6dc2b84-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5hm25\" (UID: \"c07696c4-7e34-4ad7-909d-c094f6dc2b84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"
Sep 10 17:28:43.194141 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.194121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnttw\" (UniqueName: \"kubernetes.io/projected/c07696c4-7e34-4ad7-909d-c094f6dc2b84-kube-api-access-gnttw\") pod \"cert-manager-cainjector-559cdd8c87-5hm25\" (UID: \"c07696c4-7e34-4ad7-909d-c094f6dc2b84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"
Sep 10 17:28:43.293646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.293617    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"
Sep 10 17:28:43.561869 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.561837    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5hm25"]
Sep 10 17:28:43.564499 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:28:43.564471    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc07696c4_7e34_4ad7_909d_c094f6dc2b84.slice/crio-9db2ec4c7456da7ce34102f7d4253c713abe5aa4ef7ff8b3fcd52514566325d9 WatchSource:0}: Error finding container 9db2ec4c7456da7ce34102f7d4253c713abe5aa4ef7ff8b3fcd52514566325d9: Status 404 returned error can't find the container with id 9db2ec4c7456da7ce34102f7d4253c713abe5aa4ef7ff8b3fcd52514566325d9
Sep 10 17:28:43.884650 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:43.884569    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25" event={"ID":"c07696c4-7e34-4ad7-909d-c094f6dc2b84","Type":"ContainerStarted","Data":"9db2ec4c7456da7ce34102f7d4253c713abe5aa4ef7ff8b3fcd52514566325d9"}
Sep 10 17:28:46.898930 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:46.898830    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25" event={"ID":"c07696c4-7e34-4ad7-909d-c094f6dc2b84","Type":"ContainerStarted","Data":"9a988b63be42878953c4e3f4843552b99f3200b5cc18a9d10e34d07ce809d967"}
Sep 10 17:28:46.917970 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:46.917914    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5hm25" podStartSLOduration=1.958858673 podStartE2EDuration="4.917897774s" totalImagesPullingTime="2.959039101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:46.916116715 +0000 UTC m=+295.625710981" watchObservedRunningTime="2026-09-10 17:28:46.917897774 +0000 UTC m=+295.627492039"
Sep 10 17:28:51.688940 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:51.688908    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:28:51.689907 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:51.689889    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:28:51.692353 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:51.692337    2568 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 17:28:58.537153 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.537116    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"]
Sep 10 17:28:58.543636 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.543616    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"
Sep 10 17:28:58.546691 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.546669    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:28:58.546799 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.546672    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-n4xq5\""
Sep 10 17:28:58.547678 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.547658    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 17:28:58.550821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.550798    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"]
Sep 10 17:28:58.700019 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.699978    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mk7\" (UniqueName: \"kubernetes.io/projected/34f39f00-816e-4fde-a4d6-e462637000c9-kube-api-access-c7mk7\") pod \"openshift-lws-operator-fd8ccff4c-wrw49\" (UID: \"34f39f00-816e-4fde-a4d6-e462637000c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"
Sep 10 17:28:58.700198 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.700038    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34f39f00-816e-4fde-a4d6-e462637000c9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wrw49\" (UID: \"34f39f00-816e-4fde-a4d6-e462637000c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"
Sep 10 17:28:58.800874 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.800802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mk7\" (UniqueName: \"kubernetes.io/projected/34f39f00-816e-4fde-a4d6-e462637000c9-kube-api-access-c7mk7\") pod \"openshift-lws-operator-fd8ccff4c-wrw49\" (UID: \"34f39f00-816e-4fde-a4d6-e462637000c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"
Sep 10 17:28:58.800874 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.800857    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34f39f00-816e-4fde-a4d6-e462637000c9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wrw49\" (UID: \"34f39f00-816e-4fde-a4d6-e462637000c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"
Sep 10 17:28:58.801216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.801197    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34f39f00-816e-4fde-a4d6-e462637000c9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wrw49\" (UID: \"34f39f00-816e-4fde-a4d6-e462637000c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"
Sep 10 17:28:58.809508 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.809478    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mk7\" (UniqueName: \"kubernetes.io/projected/34f39f00-816e-4fde-a4d6-e462637000c9-kube-api-access-c7mk7\") pod \"openshift-lws-operator-fd8ccff4c-wrw49\" (UID: \"34f39f00-816e-4fde-a4d6-e462637000c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"
Sep 10 17:28:58.853507 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:58.853471    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"
Sep 10 17:28:59.123768 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:59.123687    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49"]
Sep 10 17:28:59.127006 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:28:59.126974    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34f39f00_816e_4fde_a4d6_e462637000c9.slice/crio-91b806c73070eef78134f09e3a47c1117b1294ca21ccf7b2fe9cc313f73d8a15 WatchSource:0}: Error finding container 91b806c73070eef78134f09e3a47c1117b1294ca21ccf7b2fe9cc313f73d8a15: Status 404 returned error can't find the container with id 91b806c73070eef78134f09e3a47c1117b1294ca21ccf7b2fe9cc313f73d8a15
Sep 10 17:28:59.128814 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:59.128795    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:28:59.945211 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:28:59.945170    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49" event={"ID":"34f39f00-816e-4fde-a4d6-e462637000c9","Type":"ContainerStarted","Data":"91b806c73070eef78134f09e3a47c1117b1294ca21ccf7b2fe9cc313f73d8a15"}
Sep 10 17:29:01.954346 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:01.954263    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49" event={"ID":"34f39f00-816e-4fde-a4d6-e462637000c9","Type":"ContainerStarted","Data":"837baa763b3fc87c44b57c6da37888a3ad160f2367926f7a98eee1a5ad0d4ae6"}
Sep 10 17:29:01.970767 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:01.970720    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wrw49" podStartSLOduration=1.431197869 podStartE2EDuration="3.970705733s" totalImagesPullingTime="2.539507864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:01.969835052 +0000 UTC m=+310.679429322" watchObservedRunningTime="2026-09-10 17:29:01.970705733 +0000 UTC m=+310.680299999"
Sep 10 17:29:18.090931 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.090896    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"]
Sep 10 17:29:18.094121 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.094105    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.109337 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.109312    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 17:29:18.109524 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.109506    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 17:29:18.109980 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.109962    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hzpgd\""
Sep 10 17:29:18.110724 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.110709    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 17:29:18.110946 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.110933    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 17:29:18.122477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.122454    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"]
Sep 10 17:29:18.250807 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.250766    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-tmp\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.250807 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.250807    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.251060 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.250840    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-webhook-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.251060 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.250912    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-manifests\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.251060 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.250943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5v4\" (UniqueName: \"kubernetes.io/projected/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-kube-api-access-tl5v4\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.351606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.351517    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-manifests\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.351606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.351567    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5v4\" (UniqueName: \"kubernetes.io/projected/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-kube-api-access-tl5v4\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.351606 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.351604    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-tmp\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.351822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.351623    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.351822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.351654    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-webhook-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.352006 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.351984    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-manifests\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.354021 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.353994    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-tmp\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.354133 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.354049    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-webhook-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.354172 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.354127    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.361230 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.361209    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5v4\" (UniqueName: \"kubernetes.io/projected/763ecc42-f2ff-48ff-ac06-20fa3c0b0856-kube-api-access-tl5v4\") pod \"opendatahub-operator-controller-manager-b6bb7766-xfgqm\" (UID: \"763ecc42-f2ff-48ff-ac06-20fa3c0b0856\") " pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.404131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.404091    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:18.680620 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:18.680534    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"]
Sep 10 17:29:18.683639 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:29:18.683610    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763ecc42_f2ff_48ff_ac06_20fa3c0b0856.slice/crio-c61acc0d6ae2a56f7f1e61900a94a9fb19d701e58d2686ffba70f32a50d631f9 WatchSource:0}: Error finding container c61acc0d6ae2a56f7f1e61900a94a9fb19d701e58d2686ffba70f32a50d631f9: Status 404 returned error can't find the container with id c61acc0d6ae2a56f7f1e61900a94a9fb19d701e58d2686ffba70f32a50d631f9
Sep 10 17:29:19.018071 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:19.017990    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm" event={"ID":"763ecc42-f2ff-48ff-ac06-20fa3c0b0856","Type":"ContainerStarted","Data":"c61acc0d6ae2a56f7f1e61900a94a9fb19d701e58d2686ffba70f32a50d631f9"}
Sep 10 17:29:22.034902 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:22.034869    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm" event={"ID":"763ecc42-f2ff-48ff-ac06-20fa3c0b0856","Type":"ContainerStarted","Data":"dcc4f22c62e8a8b770454d0fc57f82026d59fe12280bcca0db45cdc00972195e"}
Sep 10 17:29:23.040296 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:23.040255    2568 generic.go:356] "Generic (PLEG): container finished" podID="763ecc42-f2ff-48ff-ac06-20fa3c0b0856" containerID="dcc4f22c62e8a8b770454d0fc57f82026d59fe12280bcca0db45cdc00972195e" exitCode=0
Sep 10 17:29:23.041295 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:23.040298    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm" event={"ID":"763ecc42-f2ff-48ff-ac06-20fa3c0b0856","Type":"ContainerDied","Data":"dcc4f22c62e8a8b770454d0fc57f82026d59fe12280bcca0db45cdc00972195e"}
Sep 10 17:29:24.046620 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:24.046581    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm" event={"ID":"763ecc42-f2ff-48ff-ac06-20fa3c0b0856","Type":"ContainerStarted","Data":"9a626dac4636c5a6ee6eff4f71f282023cde25a9ec4c03d94196af73b3a1d81f"}
Sep 10 17:29:24.047107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:24.046732    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:24.070745 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:24.070674    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm" podStartSLOduration=1.633773346 podStartE2EDuration="6.070651075s" totalImagesPullingTime="3.382587569s" totalInitContainerRuntime="1.05429016s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:24.067995983 +0000 UTC m=+332.777590250" watchObservedRunningTime="2026-09-10 17:29:24.070651075 +0000 UTC m=+332.780245342"
Sep 10 17:29:35.053041 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:35.053009    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b6bb7766-xfgqm"
Sep 10 17:29:38.100911 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.100878    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"]
Sep 10 17:29:38.112982 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.112956    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.116393 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.116367    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 17:29:38.117513 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.117484    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tqw57\""
Sep 10 17:29:38.117619 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.117529    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 17:29:38.117619 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.117560    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 17:29:38.117619 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.117575    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 17:29:38.124477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.124449    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"]
Sep 10 17:29:38.194070 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.194028    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tls-certs\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.194272 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.194080    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tmp\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.194272 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.194161    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djc56\" (UniqueName: \"kubernetes.io/projected/1654d45e-f4d6-4a2f-9c4c-f72633158a71-kube-api-access-djc56\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.295113 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.295078    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tls-certs\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.295223 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.295124    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tmp\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.295223 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.295161    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djc56\" (UniqueName: \"kubernetes.io/projected/1654d45e-f4d6-4a2f-9c4c-f72633158a71-kube-api-access-djc56\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.295314 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:29:38.295222    2568 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 17:29:38.295314 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:29:38.295307    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tls-certs podName:1654d45e-f4d6-4a2f-9c4c-f72633158a71 nodeName:}" failed. No retries permitted until 2026-09-10 17:29:38.795290518 +0000 UTC m=+347.504884763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tls-certs") pod "kube-auth-proxy-7bf54bbc88-h467l" (UID: "1654d45e-f4d6-4a2f-9c4c-f72633158a71") : secret "kube-auth-proxy-tls" not found
Sep 10 17:29:38.297422 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.297394    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tmp\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.307289 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.307266    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djc56\" (UniqueName: \"kubernetes.io/projected/1654d45e-f4d6-4a2f-9c4c-f72633158a71-kube-api-access-djc56\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.799938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.799896    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tls-certs\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:38.802447 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:38.802422    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1654d45e-f4d6-4a2f-9c4c-f72633158a71-tls-certs\") pod \"kube-auth-proxy-7bf54bbc88-h467l\" (UID: \"1654d45e-f4d6-4a2f-9c4c-f72633158a71\") " pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:39.022334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.022290    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"
Sep 10 17:29:39.316842 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.316810    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l"]
Sep 10 17:29:39.318565 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:29:39.318534    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1654d45e_f4d6_4a2f_9c4c_f72633158a71.slice/crio-eec0f51154a0ec1029a998f8275cebbcc12e0940c5bbc75c703f4971ae93b90d WatchSource:0}: Error finding container eec0f51154a0ec1029a998f8275cebbcc12e0940c5bbc75c703f4971ae93b90d: Status 404 returned error can't find the container with id eec0f51154a0ec1029a998f8275cebbcc12e0940c5bbc75c703f4971ae93b90d
Sep 10 17:29:39.546140 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.546099    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"]
Sep 10 17:29:39.550909 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.550885    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.554617 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.554596    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 17:29:39.554723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.554596    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 17:29:39.554723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.554715    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vr4tj\""
Sep 10 17:29:39.554800 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.554604    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 17:29:39.554855 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.554840    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 17:29:39.566944 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.566918    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"]
Sep 10 17:29:39.606572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.606478    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a8271f-6d95-40d1-948c-3621725563e5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.606572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.606518    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.606572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.606544    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.606784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.606600    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.606784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.606639    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghsj2\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-kube-api-access-ghsj2\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.606784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.606716    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.606784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.606748    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.707959 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.707917    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghsj2\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-kube-api-access-ghsj2\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.708160 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.707969    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.708160 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.707992    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.708160 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.708031    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a8271f-6d95-40d1-948c-3621725563e5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.708160 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.708045    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.708160 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.708061    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.708160 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.708078    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.708797 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.708765    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.710494 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.710463    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a8271f-6d95-40d1-948c-3621725563e5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.710584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.710472    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.710796 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.710779    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.710844 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.710825    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.716944 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.716922    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.717084 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.717065    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghsj2\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-kube-api-access-ghsj2\") pod \"istiod-openshift-gateway-7b7995d5cd-l9wz9\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:39.860481 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:39.860394    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:40.111628 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:40.111536    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l" event={"ID":"1654d45e-f4d6-4a2f-9c4c-f72633158a71","Type":"ContainerStarted","Data":"eec0f51154a0ec1029a998f8275cebbcc12e0940c5bbc75c703f4971ae93b90d"}
Sep 10 17:29:40.137110 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:40.137070    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"]
Sep 10 17:29:40.140789 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:29:40.140749    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a8271f_6d95_40d1_948c_3621725563e5.slice/crio-5d406610029e162e705b182d5f5236573fef4b40f4a2b940d8241d5c35e76e39 WatchSource:0}: Error finding container 5d406610029e162e705b182d5f5236573fef4b40f4a2b940d8241d5c35e76e39: Status 404 returned error can't find the container with id 5d406610029e162e705b182d5f5236573fef4b40f4a2b940d8241d5c35e76e39
Sep 10 17:29:41.121642 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:41.121593    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9" event={"ID":"51a8271f-6d95-40d1-948c-3621725563e5","Type":"ContainerStarted","Data":"5d406610029e162e705b182d5f5236573fef4b40f4a2b940d8241d5c35e76e39"}
Sep 10 17:29:43.132131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:43.132093    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l" event={"ID":"1654d45e-f4d6-4a2f-9c4c-f72633158a71","Type":"ContainerStarted","Data":"9b31ed406b89a6d4d077cbd8ca9bdfa9e6fdbbedb97a4cf17012fc38360fdad4"}
Sep 10 17:29:43.150946 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:43.150892    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bf54bbc88-h467l" podStartSLOduration=1.8578902369999999 podStartE2EDuration="5.15086983s" totalImagesPullingTime="3.292979593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:43.150524449 +0000 UTC m=+351.860118715" watchObservedRunningTime="2026-09-10 17:29:43.15086983 +0000 UTC m=+351.860464093"
Sep 10 17:29:44.358955 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:44.358909    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 17:29:44.359257 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:44.358988    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 17:29:45.146517 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:45.145345    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9" event={"ID":"51a8271f-6d95-40d1-948c-3621725563e5","Type":"ContainerStarted","Data":"553a487c9949a6dfaa2807facd6b43d972faea8901e8825285c18d4de8fbdff7"}
Sep 10 17:29:45.146517 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:45.146458    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:45.148449 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:45.148425    2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-l9wz9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 17:29:45.148656 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:45.148636    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9" podUID="51a8271f-6d95-40d1-948c-3621725563e5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 17:29:45.183289 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:45.183208    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9" podStartSLOduration=1.967660467 podStartE2EDuration="6.183184963s" totalImagesPullingTime="4.215524496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:45.182928533 +0000 UTC m=+353.892522800" watchObservedRunningTime="2026-09-10 17:29:45.183184963 +0000 UTC m=+353.892779231"
Sep 10 17:29:46.150270 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:46.150227    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:29:48.922312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:48.922275    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9z5gj"]
Sep 10 17:29:48.926782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:48.926759    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:48.929617 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:48.929596    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2w5pb\""
Sep 10 17:29:48.929731 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:48.929645    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 17:29:48.936624 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:48.936599    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9z5gj"]
Sep 10 17:29:48.984423 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:48.984389    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59c7cdb-b7b3-4acb-b907-76dee58922f2-tls-certs\") pod \"model-serving-api-54d89bd5fc-9z5gj\" (UID: \"b59c7cdb-b7b3-4acb-b907-76dee58922f2\") " pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:48.984423 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:48.984422    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66ml\" (UniqueName: \"kubernetes.io/projected/b59c7cdb-b7b3-4acb-b907-76dee58922f2-kube-api-access-t66ml\") pod \"model-serving-api-54d89bd5fc-9z5gj\" (UID: \"b59c7cdb-b7b3-4acb-b907-76dee58922f2\") " pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:49.085128 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:49.085097    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59c7cdb-b7b3-4acb-b907-76dee58922f2-tls-certs\") pod \"model-serving-api-54d89bd5fc-9z5gj\" (UID: \"b59c7cdb-b7b3-4acb-b907-76dee58922f2\") " pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:49.085342 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:49.085135    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66ml\" (UniqueName: \"kubernetes.io/projected/b59c7cdb-b7b3-4acb-b907-76dee58922f2-kube-api-access-t66ml\") pod \"model-serving-api-54d89bd5fc-9z5gj\" (UID: \"b59c7cdb-b7b3-4acb-b907-76dee58922f2\") " pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:49.087720 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:49.087692    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59c7cdb-b7b3-4acb-b907-76dee58922f2-tls-certs\") pod \"model-serving-api-54d89bd5fc-9z5gj\" (UID: \"b59c7cdb-b7b3-4acb-b907-76dee58922f2\") " pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:49.094882 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:49.094848    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66ml\" (UniqueName: \"kubernetes.io/projected/b59c7cdb-b7b3-4acb-b907-76dee58922f2-kube-api-access-t66ml\") pod \"model-serving-api-54d89bd5fc-9z5gj\" (UID: \"b59c7cdb-b7b3-4acb-b907-76dee58922f2\") " pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:49.237889 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:49.237807    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:49.523280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:49.520253    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9z5gj"]
Sep 10 17:29:49.525349 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:29:49.524950    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59c7cdb_b7b3_4acb_b907_76dee58922f2.slice/crio-0a68bfe4824cfa1adbc8b86a6b54874451c9bf567cc070858fb7862a29a6eda4 WatchSource:0}: Error finding container 0a68bfe4824cfa1adbc8b86a6b54874451c9bf567cc070858fb7862a29a6eda4: Status 404 returned error can't find the container with id 0a68bfe4824cfa1adbc8b86a6b54874451c9bf567cc070858fb7862a29a6eda4
Sep 10 17:29:50.170813 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:50.170766    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj" event={"ID":"b59c7cdb-b7b3-4acb-b907-76dee58922f2","Type":"ContainerStarted","Data":"0a68bfe4824cfa1adbc8b86a6b54874451c9bf567cc070858fb7862a29a6eda4"}
Sep 10 17:29:52.181574 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:52.181531    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj" event={"ID":"b59c7cdb-b7b3-4acb-b907-76dee58922f2","Type":"ContainerStarted","Data":"0ec438513dd0f31d05fc62950c15a7e5ec9c178ee63af0f54f9c97696485d43f"}
Sep 10 17:29:52.181992 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:52.181684    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:29:52.204628 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:52.204575    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj" podStartSLOduration=2.40219492 podStartE2EDuration="4.204557197s" totalImagesPullingTime="1.802362277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:52.20244501 +0000 UTC m=+360.912039275" watchObservedRunningTime="2026-09-10 17:29:52.204557197 +0000 UTC m=+360.914151462"
Sep 10 17:29:58.607375 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.607333    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"]
Sep 10 17:29:58.614750 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.614722    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:58.619678 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.619652    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 17:29:58.619938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.619921    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4vglg\""
Sep 10 17:29:58.620546 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.620525    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"]
Sep 10 17:29:58.622742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.622722    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:58.626416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.626393    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sh6df\""
Sep 10 17:29:58.626521 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.626398    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 17:29:58.629265 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.629230    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"]
Sep 10 17:29:58.637536 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.637510    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"]
Sep 10 17:29:58.668008 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.667969    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mpz5\" (UniqueName: \"kubernetes.io/projected/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-kube-api-access-9mpz5\") pod \"kserve-controller-manager-d6f4b65b8-lrltm\" (UID: \"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:58.668227 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.668119    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-cert\") pod \"kserve-controller-manager-d6f4b65b8-lrltm\" (UID: \"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:58.768842 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.768796    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skldc\" (UniqueName: \"kubernetes.io/projected/2e7225d6-da90-4e22-88ba-d5bb2556cf79-kube-api-access-skldc\") pod \"llmisvc-controller-manager-777c6bd7d5-4zr7p\" (UID: \"2e7225d6-da90-4e22-88ba-d5bb2556cf79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:58.769013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.768871    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpz5\" (UniqueName: \"kubernetes.io/projected/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-kube-api-access-9mpz5\") pod \"kserve-controller-manager-d6f4b65b8-lrltm\" (UID: \"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:58.769013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.768906    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e7225d6-da90-4e22-88ba-d5bb2556cf79-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4zr7p\" (UID: \"2e7225d6-da90-4e22-88ba-d5bb2556cf79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:58.769013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.768958    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-cert\") pod \"kserve-controller-manager-d6f4b65b8-lrltm\" (UID: \"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:58.769116 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:29:58.769053    2568 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 17:29:58.769116 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:29:58.769104    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-cert podName:ac39f1ad-5fd6-4552-93c9-ffd44795cd5a nodeName:}" failed. No retries permitted until 2026-09-10 17:29:59.26908885 +0000 UTC m=+367.978683094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-cert") pod "kserve-controller-manager-d6f4b65b8-lrltm" (UID: "ac39f1ad-5fd6-4552-93c9-ffd44795cd5a") : secret "kserve-webhook-server-cert" not found
Sep 10 17:29:58.778355 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.778331    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpz5\" (UniqueName: \"kubernetes.io/projected/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-kube-api-access-9mpz5\") pod \"kserve-controller-manager-d6f4b65b8-lrltm\" (UID: \"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:58.869687 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.869585    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e7225d6-da90-4e22-88ba-d5bb2556cf79-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4zr7p\" (UID: \"2e7225d6-da90-4e22-88ba-d5bb2556cf79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:58.869850 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.869734    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skldc\" (UniqueName: \"kubernetes.io/projected/2e7225d6-da90-4e22-88ba-d5bb2556cf79-kube-api-access-skldc\") pod \"llmisvc-controller-manager-777c6bd7d5-4zr7p\" (UID: \"2e7225d6-da90-4e22-88ba-d5bb2556cf79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:58.869850 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:29:58.869767    2568 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 10 17:29:58.870188 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:29:58.869853    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7225d6-da90-4e22-88ba-d5bb2556cf79-cert podName:2e7225d6-da90-4e22-88ba-d5bb2556cf79 nodeName:}" failed. No retries permitted until 2026-09-10 17:29:59.369829731 +0000 UTC m=+368.079423988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e7225d6-da90-4e22-88ba-d5bb2556cf79-cert") pod "llmisvc-controller-manager-777c6bd7d5-4zr7p" (UID: "2e7225d6-da90-4e22-88ba-d5bb2556cf79") : secret "llmisvc-webhook-server-cert" not found
Sep 10 17:29:58.879135 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:58.879112    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skldc\" (UniqueName: \"kubernetes.io/projected/2e7225d6-da90-4e22-88ba-d5bb2556cf79-kube-api-access-skldc\") pod \"llmisvc-controller-manager-777c6bd7d5-4zr7p\" (UID: \"2e7225d6-da90-4e22-88ba-d5bb2556cf79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:59.272653 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:59.272622    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-cert\") pod \"kserve-controller-manager-d6f4b65b8-lrltm\" (UID: \"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:59.275036 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:59.275011    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac39f1ad-5fd6-4552-93c9-ffd44795cd5a-cert\") pod \"kserve-controller-manager-d6f4b65b8-lrltm\" (UID: \"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:59.373195 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:59.373156    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e7225d6-da90-4e22-88ba-d5bb2556cf79-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4zr7p\" (UID: \"2e7225d6-da90-4e22-88ba-d5bb2556cf79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:59.375709 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:59.375684    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e7225d6-da90-4e22-88ba-d5bb2556cf79-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4zr7p\" (UID: \"2e7225d6-da90-4e22-88ba-d5bb2556cf79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:59.526384 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:59.526298    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:29:59.535383 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:59.535356    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:29:59.806673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:59.806553    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"]
Sep 10 17:29:59.809408 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:29:59.809365    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac39f1ad_5fd6_4552_93c9_ffd44795cd5a.slice/crio-f4742db2ce10279a8e04c1305eac94e46e387e0ffad43d2d12c272e42cf76953 WatchSource:0}: Error finding container f4742db2ce10279a8e04c1305eac94e46e387e0ffad43d2d12c272e42cf76953: Status 404 returned error can't find the container with id f4742db2ce10279a8e04c1305eac94e46e387e0ffad43d2d12c272e42cf76953
Sep 10 17:29:59.814522 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:29:59.814486    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"]
Sep 10 17:29:59.819457 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:29:59.819426    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2e7225d6_da90_4e22_88ba_d5bb2556cf79.slice/crio-152507084b642dec35f82472cd8e1565a822d82fb4b7ccbd247a3559532fe919 WatchSource:0}: Error finding container 152507084b642dec35f82472cd8e1565a822d82fb4b7ccbd247a3559532fe919: Status 404 returned error can't find the container with id 152507084b642dec35f82472cd8e1565a822d82fb4b7ccbd247a3559532fe919
Sep 10 17:30:00.025866 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.025837    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"]
Sep 10 17:30:00.026069 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.026032    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9" podUID="51a8271f-6d95-40d1-948c-3621725563e5" containerName="discovery" containerID="cri-o://553a487c9949a6dfaa2807facd6b43d972faea8901e8825285c18d4de8fbdff7" gracePeriod=30
Sep 10 17:30:00.222623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.222589    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p" event={"ID":"2e7225d6-da90-4e22-88ba-d5bb2556cf79","Type":"ContainerStarted","Data":"152507084b642dec35f82472cd8e1565a822d82fb4b7ccbd247a3559532fe919"}
Sep 10 17:30:00.224284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.224259    2568 generic.go:356] "Generic (PLEG): container finished" podID="51a8271f-6d95-40d1-948c-3621725563e5" containerID="553a487c9949a6dfaa2807facd6b43d972faea8901e8825285c18d4de8fbdff7" exitCode=0
Sep 10 17:30:00.224401 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.224334    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9" event={"ID":"51a8271f-6d95-40d1-948c-3621725563e5","Type":"ContainerDied","Data":"553a487c9949a6dfaa2807facd6b43d972faea8901e8825285c18d4de8fbdff7"}
Sep 10 17:30:00.225571 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.225545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm" event={"ID":"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a","Type":"ContainerStarted","Data":"f4742db2ce10279a8e04c1305eac94e46e387e0ffad43d2d12c272e42cf76953"}
Sep 10 17:30:00.257178 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.257154    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:30:00.382440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.382348    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a8271f-6d95-40d1-948c-3621725563e5-local-certs\") pod \"51a8271f-6d95-40d1-948c-3621725563e5\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") "
Sep 10 17:30:00.382440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.382420    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghsj2\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-kube-api-access-ghsj2\") pod \"51a8271f-6d95-40d1-948c-3621725563e5\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") "
Sep 10 17:30:00.382673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.382451    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-dns-cert\") pod \"51a8271f-6d95-40d1-948c-3621725563e5\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") "
Sep 10 17:30:00.382673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.382483    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-kubeconfig\") pod \"51a8271f-6d95-40d1-948c-3621725563e5\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") "
Sep 10 17:30:00.382673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.382509    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-ca-configmap\") pod \"51a8271f-6d95-40d1-948c-3621725563e5\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") "
Sep 10 17:30:00.382673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.382549    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-cacerts\") pod \"51a8271f-6d95-40d1-948c-3621725563e5\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") "
Sep 10 17:30:00.382673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.382610    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-istio-token\") pod \"51a8271f-6d95-40d1-948c-3621725563e5\" (UID: \"51a8271f-6d95-40d1-948c-3621725563e5\") "
Sep 10 17:30:00.383265 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.383149    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "51a8271f-6d95-40d1-948c-3621725563e5" (UID: "51a8271f-6d95-40d1-948c-3621725563e5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:30:00.385302 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.385221    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "51a8271f-6d95-40d1-948c-3621725563e5" (UID: "51a8271f-6d95-40d1-948c-3621725563e5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:30:00.385434 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.385376    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-kube-api-access-ghsj2" (OuterVolumeSpecName: "kube-api-access-ghsj2") pod "51a8271f-6d95-40d1-948c-3621725563e5" (UID: "51a8271f-6d95-40d1-948c-3621725563e5"). InnerVolumeSpecName "kube-api-access-ghsj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:30:00.386401 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.386354    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "51a8271f-6d95-40d1-948c-3621725563e5" (UID: "51a8271f-6d95-40d1-948c-3621725563e5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:30:00.386401 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.386367    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a8271f-6d95-40d1-948c-3621725563e5-local-certs" (OuterVolumeSpecName: "local-certs") pod "51a8271f-6d95-40d1-948c-3621725563e5" (UID: "51a8271f-6d95-40d1-948c-3621725563e5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:00.386554 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.386424    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "51a8271f-6d95-40d1-948c-3621725563e5" (UID: "51a8271f-6d95-40d1-948c-3621725563e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:30:00.387093 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.387069    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-cacerts" (OuterVolumeSpecName: "cacerts") pod "51a8271f-6d95-40d1-948c-3621725563e5" (UID: "51a8271f-6d95-40d1-948c-3621725563e5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:30:00.483938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.483813    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-istio-token\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:30:00.483938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.483846    2568 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a8271f-6d95-40d1-948c-3621725563e5-local-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:30:00.483938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.483860    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghsj2\" (UniqueName: \"kubernetes.io/projected/51a8271f-6d95-40d1-948c-3621725563e5-kube-api-access-ghsj2\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:30:00.483938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.483875    2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-dns-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:30:00.483938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.483889    2568 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-istio-kubeconfig\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:30:00.483938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.483902    2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51a8271f-6d95-40d1-948c-3621725563e5-istio-csr-ca-configmap\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:30:00.483938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:00.483914    2568 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51a8271f-6d95-40d1-948c-3621725563e5-cacerts\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:30:01.233156 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:01.233076    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"
Sep 10 17:30:01.233156 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:01.233104    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9" event={"ID":"51a8271f-6d95-40d1-948c-3621725563e5","Type":"ContainerDied","Data":"5d406610029e162e705b182d5f5236573fef4b40f4a2b940d8241d5c35e76e39"}
Sep 10 17:30:01.233645 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:01.233206    2568 scope.go:117] "RemoveContainer" containerID="553a487c9949a6dfaa2807facd6b43d972faea8901e8825285c18d4de8fbdff7"
Sep 10 17:30:01.258282 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:01.258258    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"]
Sep 10 17:30:01.268322 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:01.268293    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l9wz9"]
Sep 10 17:30:01.763904 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:01.763867    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a8271f-6d95-40d1-948c-3621725563e5" path="/var/lib/kubelet/pods/51a8271f-6d95-40d1-948c-3621725563e5/volumes"
Sep 10 17:30:03.191050 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:03.190971    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9z5gj"
Sep 10 17:30:03.245472 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:03.245438    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p" event={"ID":"2e7225d6-da90-4e22-88ba-d5bb2556cf79","Type":"ContainerStarted","Data":"52d481499967c87d95990a2a673a520753970c5f52f76db5956ca833a0665290"}
Sep 10 17:30:03.245656 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:03.245548    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:30:03.247348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:03.247313    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm" event={"ID":"ac39f1ad-5fd6-4552-93c9-ffd44795cd5a","Type":"ContainerStarted","Data":"b0ecf1a39a5a4909a371fea37f57b13afefaafc255f3d2d4289a965d3fd528f9"}
Sep 10 17:30:03.247485 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:03.247469    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:30:03.284673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:03.284615    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p" podStartSLOduration=2.164285392 podStartE2EDuration="5.284597617s" totalImagesPullingTime="3.120312225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:03.2826285 +0000 UTC m=+371.992222766" watchObservedRunningTime="2026-09-10 17:30:03.284597617 +0000 UTC m=+371.994191930"
Sep 10 17:30:03.300431 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:03.300376    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm" podStartSLOduration=2.921391119 podStartE2EDuration="5.300359407s" totalImagesPullingTime="2.378968288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:03.299083991 +0000 UTC m=+372.008678257" watchObservedRunningTime="2026-09-10 17:30:03.300359407 +0000 UTC m=+372.009953673"
Sep 10 17:30:34.253389 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:34.253356    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zr7p"
Sep 10 17:30:34.256341 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:30:34.256319    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lrltm"
Sep 10 17:31:21.839131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.839050    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6556d7688-dch9k"]
Sep 10 17:31:21.839566 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.839387    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a8271f-6d95-40d1-948c-3621725563e5" containerName="discovery"
Sep 10 17:31:21.839566 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.839399    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a8271f-6d95-40d1-948c-3621725563e5" containerName="discovery"
Sep 10 17:31:21.839566 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.839451    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a8271f-6d95-40d1-948c-3621725563e5" containerName="discovery"
Sep 10 17:31:21.841502 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.841469    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:21.858077 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.858049    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6556d7688-dch9k"]
Sep 10 17:31:21.933821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.933781    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8edf05-b178-4a63-9548-42707aca3656-console-serving-cert\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:21.933988 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.933834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-trusted-ca-bundle\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:21.933988 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.933853    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-oauth-serving-cert\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:21.933988 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.933936    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-console-config\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:21.933988 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.933984    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e8edf05-b178-4a63-9548-42707aca3656-console-oauth-config\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:21.934147 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.934002    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-service-ca\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:21.934147 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:21.934033    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb8hh\" (UniqueName: \"kubernetes.io/projected/3e8edf05-b178-4a63-9548-42707aca3656-kube-api-access-wb8hh\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.035381 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.035344    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-console-config\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.035572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.035393    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e8edf05-b178-4a63-9548-42707aca3656-console-oauth-config\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.035572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.035411    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-service-ca\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.035572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.035437    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8hh\" (UniqueName: \"kubernetes.io/projected/3e8edf05-b178-4a63-9548-42707aca3656-kube-api-access-wb8hh\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.035572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.035459    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8edf05-b178-4a63-9548-42707aca3656-console-serving-cert\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.035572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.035495    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-trusted-ca-bundle\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.035572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.035512    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-oauth-serving-cert\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.036153 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.036127    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-console-config\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.036260 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.036195    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-service-ca\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.036260 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.036250    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-oauth-serving-cert\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.036367 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.036351    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8edf05-b178-4a63-9548-42707aca3656-trusted-ca-bundle\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.037818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.037791    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8edf05-b178-4a63-9548-42707aca3656-console-serving-cert\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.037940 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.037918    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e8edf05-b178-4a63-9548-42707aca3656-console-oauth-config\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.044948 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.044928    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8hh\" (UniqueName: \"kubernetes.io/projected/3e8edf05-b178-4a63-9548-42707aca3656-kube-api-access-wb8hh\") pod \"console-6556d7688-dch9k\" (UID: \"3e8edf05-b178-4a63-9548-42707aca3656\") " pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.150452 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.150349    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:22.424956 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.422482    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6556d7688-dch9k"]
Sep 10 17:31:22.427717 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:31:22.427686    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8edf05_b178_4a63_9548_42707aca3656.slice/crio-c1af2da83c738279144cc990449048c8696764b9079edd035d2dd6561271e759 WatchSource:0}: Error finding container c1af2da83c738279144cc990449048c8696764b9079edd035d2dd6561271e759: Status 404 returned error can't find the container with id c1af2da83c738279144cc990449048c8696764b9079edd035d2dd6561271e759
Sep 10 17:31:22.584623 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.584588    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6556d7688-dch9k" event={"ID":"3e8edf05-b178-4a63-9548-42707aca3656","Type":"ContainerStarted","Data":"72d89eacb5359bcc4f7d32d3e99179bc2fe27b1e3a06a5da5fbad8e78b0d00de"}
Sep 10 17:31:22.584788 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.584642    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6556d7688-dch9k" event={"ID":"3e8edf05-b178-4a63-9548-42707aca3656","Type":"ContainerStarted","Data":"c1af2da83c738279144cc990449048c8696764b9079edd035d2dd6561271e759"}
Sep 10 17:31:22.607978 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:22.607924    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6556d7688-dch9k" podStartSLOduration=1.60790755 podStartE2EDuration="1.60790755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:22.606518459 +0000 UTC m=+451.316112725" watchObservedRunningTime="2026-09-10 17:31:22.60790755 +0000 UTC m=+451.317501815"
Sep 10 17:31:28.737047 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:28.737010    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"]
Sep 10 17:31:28.740276 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:28.740260    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:28.743202 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:28.743174    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 17:31:28.744317 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:28.744289    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 17:31:28.744459 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:28.744438    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pc895\""
Sep 10 17:31:28.749834 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:28.749815    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"]
Sep 10 17:31:28.894747 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:28.894695    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9k55\" (UniqueName: \"kubernetes.io/projected/21b2b526-2204-4185-a032-846fd9132aa5-kube-api-access-r9k55\") pod \"limitador-operator-controller-manager-74495c69f8-smpb7\" (UID: \"21b2b526-2204-4185-a032-846fd9132aa5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:28.995996 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:28.995897    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9k55\" (UniqueName: \"kubernetes.io/projected/21b2b526-2204-4185-a032-846fd9132aa5-kube-api-access-r9k55\") pod \"limitador-operator-controller-manager-74495c69f8-smpb7\" (UID: \"21b2b526-2204-4185-a032-846fd9132aa5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:29.011712 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:29.011681    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9k55\" (UniqueName: \"kubernetes.io/projected/21b2b526-2204-4185-a032-846fd9132aa5-kube-api-access-r9k55\") pod \"limitador-operator-controller-manager-74495c69f8-smpb7\" (UID: \"21b2b526-2204-4185-a032-846fd9132aa5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:29.051598 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:29.051560    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:29.338136 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:29.338110    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"]
Sep 10 17:31:29.340771 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:31:29.340745    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b2b526_2204_4185_a032_846fd9132aa5.slice/crio-28024ea3b281e4b82d4cd40fa296a1044b2b40cf75224cb2a44fef7e9b75b447 WatchSource:0}: Error finding container 28024ea3b281e4b82d4cd40fa296a1044b2b40cf75224cb2a44fef7e9b75b447: Status 404 returned error can't find the container with id 28024ea3b281e4b82d4cd40fa296a1044b2b40cf75224cb2a44fef7e9b75b447
Sep 10 17:31:29.615582 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:29.615505    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7" event={"ID":"21b2b526-2204-4185-a032-846fd9132aa5","Type":"ContainerStarted","Data":"28024ea3b281e4b82d4cd40fa296a1044b2b40cf75224cb2a44fef7e9b75b447"}
Sep 10 17:31:32.150802 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:32.150755    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:32.151326 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:32.150835    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:32.156273 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:32.156246    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:32.631292 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:32.631250    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7" event={"ID":"21b2b526-2204-4185-a032-846fd9132aa5","Type":"ContainerStarted","Data":"8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da"}
Sep 10 17:31:32.631536 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:32.631514    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:32.635056 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:32.635028    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6556d7688-dch9k"
Sep 10 17:31:32.655446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:32.655394    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7" podStartSLOduration=2.270750268 podStartE2EDuration="4.655378887s" totalImagesPullingTime="2.384628619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:32.65395777 +0000 UTC m=+461.363552035" watchObservedRunningTime="2026-09-10 17:31:32.655378887 +0000 UTC m=+461.364973185"
Sep 10 17:31:32.709993 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:32.709959    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78f7ccd44-5c57j"]
Sep 10 17:31:33.055625 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.055588    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"]
Sep 10 17:31:33.058946 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.058922    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:33.061711 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.061686    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5qmxl\""
Sep 10 17:31:33.074255 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.074217    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"]
Sep 10 17:31:33.130794 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.130761    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" (UID: \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:33.130977 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.130828    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvp84\" (UniqueName: \"kubernetes.io/projected/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-kube-api-access-zvp84\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" (UID: \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:33.232031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.231988    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvp84\" (UniqueName: \"kubernetes.io/projected/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-kube-api-access-zvp84\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" (UID: \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:33.232530 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.232087    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" (UID: \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:33.232530 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.232501    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" (UID: \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:33.241481 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.241454    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvp84\" (UniqueName: \"kubernetes.io/projected/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-kube-api-access-zvp84\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" (UID: \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:33.369631 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.369541    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:33.647466 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:33.647437    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"]
Sep 10 17:31:33.649517 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:31:33.649486    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ad6dd7_3b1c_459b_b82e_3ffefa735b92.slice/crio-3f8e2a84e4ec8d0176fb88606fe8848da3a31ec7e4eded77ee59ffafa8717b73 WatchSource:0}: Error finding container 3f8e2a84e4ec8d0176fb88606fe8848da3a31ec7e4eded77ee59ffafa8717b73: Status 404 returned error can't find the container with id 3f8e2a84e4ec8d0176fb88606fe8848da3a31ec7e4eded77ee59ffafa8717b73
Sep 10 17:31:34.644789 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:34.644749    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd" event={"ID":"44ad6dd7-3b1c-459b-b82e-3ffefa735b92","Type":"ContainerStarted","Data":"3f8e2a84e4ec8d0176fb88606fe8848da3a31ec7e4eded77ee59ffafa8717b73"}
Sep 10 17:31:35.595405 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:35.595372    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-887dw"]
Sep 10 17:31:35.598679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:35.598650    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-887dw"
Sep 10 17:31:35.606380 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:35.606331    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7fgjt\""
Sep 10 17:31:35.618562 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:35.618533    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-887dw"]
Sep 10 17:31:35.653798 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:35.653761    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdzb\" (UniqueName: \"kubernetes.io/projected/b0a3af90-b4f8-4610-8c85-c2e6603d1a67-kube-api-access-hsdzb\") pod \"authorino-operator-76b478bd78-887dw\" (UID: \"b0a3af90-b4f8-4610-8c85-c2e6603d1a67\") " pod="kuadrant-system/authorino-operator-76b478bd78-887dw"
Sep 10 17:31:35.754497 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:35.754436    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdzb\" (UniqueName: \"kubernetes.io/projected/b0a3af90-b4f8-4610-8c85-c2e6603d1a67-kube-api-access-hsdzb\") pod \"authorino-operator-76b478bd78-887dw\" (UID: \"b0a3af90-b4f8-4610-8c85-c2e6603d1a67\") " pod="kuadrant-system/authorino-operator-76b478bd78-887dw"
Sep 10 17:31:35.775556 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:35.775528    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdzb\" (UniqueName: \"kubernetes.io/projected/b0a3af90-b4f8-4610-8c85-c2e6603d1a67-kube-api-access-hsdzb\") pod \"authorino-operator-76b478bd78-887dw\" (UID: \"b0a3af90-b4f8-4610-8c85-c2e6603d1a67\") " pod="kuadrant-system/authorino-operator-76b478bd78-887dw"
Sep 10 17:31:35.912465 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:35.912378    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-887dw"
Sep 10 17:31:36.195162 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:36.195027    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-887dw"]
Sep 10 17:31:37.345912 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:31:37.345880    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a3af90_b4f8_4610_8c85_c2e6603d1a67.slice/crio-77e530be141aae47450a9906782ec7769685369be2fa312dcdeb2d004dea069c WatchSource:0}: Error finding container 77e530be141aae47450a9906782ec7769685369be2fa312dcdeb2d004dea069c: Status 404 returned error can't find the container with id 77e530be141aae47450a9906782ec7769685369be2fa312dcdeb2d004dea069c
Sep 10 17:31:37.662530 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.662418    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd" event={"ID":"44ad6dd7-3b1c-459b-b82e-3ffefa735b92","Type":"ContainerStarted","Data":"b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68"}
Sep 10 17:31:37.662688 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.662596    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:37.664102 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.664065    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-887dw" event={"ID":"b0a3af90-b4f8-4610-8c85-c2e6603d1a67","Type":"ContainerStarted","Data":"77e530be141aae47450a9906782ec7769685369be2fa312dcdeb2d004dea069c"}
Sep 10 17:31:37.692609 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.692528    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd" podStartSLOduration=0.932810891 podStartE2EDuration="4.692508167s" totalImagesPullingTime="3.759697276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:37.690951767 +0000 UTC m=+466.400546033" watchObservedRunningTime="2026-09-10 17:31:37.692508167 +0000 UTC m=+466.402102436"
Sep 10 17:31:37.967994 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.967894    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"]
Sep 10 17:31:37.971335 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.971315    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"
Sep 10 17:31:37.974368 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.974343    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pxj54\""
Sep 10 17:31:37.974695 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.974367    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 17:31:37.985980 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:37.985951    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"]
Sep 10 17:31:38.072679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:38.072636    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54x7x\" (UniqueName: \"kubernetes.io/projected/9acead52-cfd6-48ae-9d7e-89cc1e472a93-kube-api-access-54x7x\") pod \"dns-operator-controller-manager-7b84cd7c9c-6qqc6\" (UID: \"9acead52-cfd6-48ae-9d7e-89cc1e472a93\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"
Sep 10 17:31:38.174047 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:38.174006    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54x7x\" (UniqueName: \"kubernetes.io/projected/9acead52-cfd6-48ae-9d7e-89cc1e472a93-kube-api-access-54x7x\") pod \"dns-operator-controller-manager-7b84cd7c9c-6qqc6\" (UID: \"9acead52-cfd6-48ae-9d7e-89cc1e472a93\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"
Sep 10 17:31:38.188832 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:38.188790    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54x7x\" (UniqueName: \"kubernetes.io/projected/9acead52-cfd6-48ae-9d7e-89cc1e472a93-kube-api-access-54x7x\") pod \"dns-operator-controller-manager-7b84cd7c9c-6qqc6\" (UID: \"9acead52-cfd6-48ae-9d7e-89cc1e472a93\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"
Sep 10 17:31:38.281360 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:38.281320    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"
Sep 10 17:31:38.592773 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:38.592683    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"]
Sep 10 17:31:38.598776 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:31:38.598749    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acead52_cfd6_48ae_9d7e_89cc1e472a93.slice/crio-d1ffafd8109652ae0560f0469d19adafae0c55fdc5e1f924836482b3dfec6771 WatchSource:0}: Error finding container d1ffafd8109652ae0560f0469d19adafae0c55fdc5e1f924836482b3dfec6771: Status 404 returned error can't find the container with id d1ffafd8109652ae0560f0469d19adafae0c55fdc5e1f924836482b3dfec6771
Sep 10 17:31:38.671188 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:38.671146    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6" event={"ID":"9acead52-cfd6-48ae-9d7e-89cc1e472a93","Type":"ContainerStarted","Data":"d1ffafd8109652ae0560f0469d19adafae0c55fdc5e1f924836482b3dfec6771"}
Sep 10 17:31:39.679007 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:39.678976    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-887dw" event={"ID":"b0a3af90-b4f8-4610-8c85-c2e6603d1a67","Type":"ContainerStarted","Data":"db8f7352a82a7147489140f1c5e88fe999f2ab2eb051b8ab13caafec0bce8b49"}
Sep 10 17:31:39.679472 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:39.679130    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-887dw"
Sep 10 17:31:39.697360 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:39.697305    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-887dw" podStartSLOduration=3.087301378 podStartE2EDuration="4.697288243s" totalImagesPullingTime="1.609986865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:39.695837486 +0000 UTC m=+468.405431763" watchObservedRunningTime="2026-09-10 17:31:39.697288243 +0000 UTC m=+468.406882508"
Sep 10 17:31:40.685576 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:40.685532    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6" event={"ID":"9acead52-cfd6-48ae-9d7e-89cc1e472a93","Type":"ContainerStarted","Data":"96dcd3780da57546cf3a8ccdd0c199049f4dd367068da36ca497595738e8c5da"}
Sep 10 17:31:40.711262 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:40.711198    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6" podStartSLOduration=2.083899045 podStartE2EDuration="3.711179761s" totalImagesPullingTime="1.627280716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:40.708938693 +0000 UTC m=+469.418532970" watchObservedRunningTime="2026-09-10 17:31:40.711179761 +0000 UTC m=+469.420774028"
Sep 10 17:31:41.689880 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:41.689853    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"
Sep 10 17:31:43.640419 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:43.640384    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:48.673496 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:48.673457    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:50.688334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.688297    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-887dw"
Sep 10 17:31:50.847763 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.847733    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"]
Sep 10 17:31:50.847968 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.847948    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" containerName="manager" containerID="cri-o://b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68" gracePeriod=2
Sep 10 17:31:50.864445 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.864421    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"]
Sep 10 17:31:50.865824 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:50.865788    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\": pods \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:50.883869 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.883831    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"]
Sep 10 17:31:50.884475 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.884451    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" containerName="manager"
Sep 10 17:31:50.884475 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.884473    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" containerName="manager"
Sep 10 17:31:50.884639 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.884553    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" containerName="manager"
Sep 10 17:31:50.887630 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.887604    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"]
Sep 10 17:31:50.887778 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.887749    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:50.888489 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.888035    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7" podUID="21b2b526-2204-4185-a032-846fd9132aa5" containerName="manager" containerID="cri-o://8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da" gracePeriod=2
Sep 10 17:31:50.889043 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.888751    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"]
Sep 10 17:31:50.889924 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:50.889877    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:50.903846 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.903819    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"]
Sep 10 17:31:50.915672 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.915642    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"]
Sep 10 17:31:50.917391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.916159    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b2b526-2204-4185-a032-846fd9132aa5" containerName="manager"
Sep 10 17:31:50.917391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.916180    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b2b526-2204-4185-a032-846fd9132aa5" containerName="manager"
Sep 10 17:31:50.917391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.916292    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b2b526-2204-4185-a032-846fd9132aa5" containerName="manager"
Sep 10 17:31:50.923559 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.923537    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"
Sep 10 17:31:50.932801 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.932772    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"]
Sep 10 17:31:50.938095 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:50.938066    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-smpb7\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="21b2b526-2204-4185-a032-846fd9132aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:50.961439 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:50.961362    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:50.987336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.987308    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bpp\" (UniqueName: \"kubernetes.io/projected/a6c3e358-9880-4d4a-af39-5752131d94b2-kube-api-access-f2bpp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7pb5v\" (UID: \"a6c3e358-9880-4d4a-af39-5752131d94b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:50.987443 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.987407    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6c3e358-9880-4d4a-af39-5752131d94b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7pb5v\" (UID: \"a6c3e358-9880-4d4a-af39-5752131d94b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:50.987534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:50.987515    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5w5z\" (UniqueName: \"kubernetes.io/projected/e357f2b7-993e-433c-bc09-5d2e65445339-kube-api-access-z5w5z\") pod \"limitador-operator-controller-manager-74495c69f8-mtmz2\" (UID: \"e357f2b7-993e-433c-bc09-5d2e65445339\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"
Sep 10 17:31:51.088930 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.088895    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5w5z\" (UniqueName: \"kubernetes.io/projected/e357f2b7-993e-433c-bc09-5d2e65445339-kube-api-access-z5w5z\") pod \"limitador-operator-controller-manager-74495c69f8-mtmz2\" (UID: \"e357f2b7-993e-433c-bc09-5d2e65445339\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"
Sep 10 17:31:51.089086 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.088953    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bpp\" (UniqueName: \"kubernetes.io/projected/a6c3e358-9880-4d4a-af39-5752131d94b2-kube-api-access-f2bpp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7pb5v\" (UID: \"a6c3e358-9880-4d4a-af39-5752131d94b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:51.089086 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.089022    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6c3e358-9880-4d4a-af39-5752131d94b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7pb5v\" (UID: \"a6c3e358-9880-4d4a-af39-5752131d94b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:51.089487 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.089457    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6c3e358-9880-4d4a-af39-5752131d94b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7pb5v\" (UID: \"a6c3e358-9880-4d4a-af39-5752131d94b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:51.108917 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.108880    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5w5z\" (UniqueName: \"kubernetes.io/projected/e357f2b7-993e-433c-bc09-5d2e65445339-kube-api-access-z5w5z\") pod \"limitador-operator-controller-manager-74495c69f8-mtmz2\" (UID: \"e357f2b7-993e-433c-bc09-5d2e65445339\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"
Sep 10 17:31:51.115196 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.115169    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bpp\" (UniqueName: \"kubernetes.io/projected/a6c3e358-9880-4d4a-af39-5752131d94b2-kube-api-access-f2bpp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7pb5v\" (UID: \"a6c3e358-9880-4d4a-af39-5752131d94b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:51.133279 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.133258    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:51.135740 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.135720    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:51.136495 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.136472    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:51.139517 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.139495    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-smpb7\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="21b2b526-2204-4185-a032-846fd9132aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:51.141624 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.141600    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-smpb7\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="21b2b526-2204-4185-a032-846fd9132aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:51.147632 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.147610    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:51.189864 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.189838    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvp84\" (UniqueName: \"kubernetes.io/projected/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-kube-api-access-zvp84\") pod \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\" (UID: \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\") "
Sep 10 17:31:51.189983 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.189897    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9k55\" (UniqueName: \"kubernetes.io/projected/21b2b526-2204-4185-a032-846fd9132aa5-kube-api-access-r9k55\") pod \"21b2b526-2204-4185-a032-846fd9132aa5\" (UID: \"21b2b526-2204-4185-a032-846fd9132aa5\") "
Sep 10 17:31:51.189983 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.189967    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-extensions-socket-volume\") pod \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\" (UID: \"44ad6dd7-3b1c-459b-b82e-3ffefa735b92\") "
Sep 10 17:31:51.190504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.190480    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "44ad6dd7-3b1c-459b-b82e-3ffefa735b92" (UID: "44ad6dd7-3b1c-459b-b82e-3ffefa735b92"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:31:51.192056 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.192031    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-kube-api-access-zvp84" (OuterVolumeSpecName: "kube-api-access-zvp84") pod "44ad6dd7-3b1c-459b-b82e-3ffefa735b92" (UID: "44ad6dd7-3b1c-459b-b82e-3ffefa735b92"). InnerVolumeSpecName "kube-api-access-zvp84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:31:51.192134 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.192077    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b2b526-2204-4185-a032-846fd9132aa5-kube-api-access-r9k55" (OuterVolumeSpecName: "kube-api-access-r9k55") pod "21b2b526-2204-4185-a032-846fd9132aa5" (UID: "21b2b526-2204-4185-a032-846fd9132aa5"). InnerVolumeSpecName "kube-api-access-r9k55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:31:51.284066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.284034    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:51.290014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.289994    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"
Sep 10 17:31:51.291350 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.291326    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9k55\" (UniqueName: \"kubernetes.io/projected/21b2b526-2204-4185-a032-846fd9132aa5-kube-api-access-r9k55\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:51.291350 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.291350    2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-extensions-socket-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:51.291474 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.291360    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvp84\" (UniqueName: \"kubernetes.io/projected/44ad6dd7-3b1c-459b-b82e-3ffefa735b92-kube-api-access-zvp84\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:51.589710 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.589683    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"]
Sep 10 17:31:51.591114 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.591093    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"]
Sep 10 17:31:51.592592 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:31:51.592560    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode357f2b7_993e_433c_bc09_5d2e65445339.slice/crio-0f255ed764b6f9f15642f83a56f6f3ebd68a94a67186431e51a6f7b85d7de192 WatchSource:0}: Error finding container 0f255ed764b6f9f15642f83a56f6f3ebd68a94a67186431e51a6f7b85d7de192: Status 404 returned error can't find the container with id 0f255ed764b6f9f15642f83a56f6f3ebd68a94a67186431e51a6f7b85d7de192
Sep 10 17:31:51.593222 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:31:51.593198    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c3e358_9880_4d4a_af39_5752131d94b2.slice/crio-1896f076d1228ee3c498c836fd0601680fde8c22a1433798d3ba8dc898e08360 WatchSource:0}: Error finding container 1896f076d1228ee3c498c836fd0601680fde8c22a1433798d3ba8dc898e08360: Status 404 returned error can't find the container with id 1896f076d1228ee3c498c836fd0601680fde8c22a1433798d3ba8dc898e08360
Sep 10 17:31:51.738123 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.738083    2568 generic.go:356] "Generic (PLEG): container finished" podID="21b2b526-2204-4185-a032-846fd9132aa5" containerID="8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da" exitCode=0
Sep 10 17:31:51.738608 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.738139    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:51.738608 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.738184    2568 scope.go:117] "RemoveContainer" containerID="8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da"
Sep 10 17:31:51.739705 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.739681    2568 generic.go:356] "Generic (PLEG): container finished" podID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" containerID="b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68" exitCode=0
Sep 10 17:31:51.739813 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.739730    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:51.740773 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.740738    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:51.742265 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.742219    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2" event={"ID":"e357f2b7-993e-433c-bc09-5d2e65445339","Type":"ContainerStarted","Data":"b0e33f4d7b98dadb7f9f0e9b9c6ad6ec1bc3091b172cedbd606db76b0e3ab57f"}
Sep 10 17:31:51.742531 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.742501    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2" event={"ID":"e357f2b7-993e-433c-bc09-5d2e65445339","Type":"ContainerStarted","Data":"0f255ed764b6f9f15642f83a56f6f3ebd68a94a67186431e51a6f7b85d7de192"}
Sep 10 17:31:51.742698 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.742685    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"
Sep 10 17:31:51.742893 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.742864    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-smpb7\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="21b2b526-2204-4185-a032-846fd9132aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:51.744695 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.744668    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v" event={"ID":"a6c3e358-9880-4d4a-af39-5752131d94b2","Type":"ContainerStarted","Data":"d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2"}
Sep 10 17:31:51.744780 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.744720    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v" event={"ID":"a6c3e358-9880-4d4a-af39-5752131d94b2","Type":"ContainerStarted","Data":"1896f076d1228ee3c498c836fd0601680fde8c22a1433798d3ba8dc898e08360"}
Sep 10 17:31:51.744780 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.744750    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:31:51.745036 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.745011    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-smpb7\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="21b2b526-2204-4185-a032-846fd9132aa5" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-smpb7"
Sep 10 17:31:51.748867 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.748848    2568 scope.go:117] "RemoveContainer" containerID="8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da"
Sep 10 17:31:51.749183 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.749164    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da\": container with ID starting with 8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da not found: ID does not exist" containerID="8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da"
Sep 10 17:31:51.749286 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.749195    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da"} err="failed to get container status \"8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da\": rpc error: code = NotFound desc = could not find container \"8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da\": container with ID starting with 8f9f04c1ddca55f3702b2a72a332841da18e44de482f10d0708075d4fb3a46da not found: ID does not exist"
Sep 10 17:31:51.749286 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.749220    2568 scope.go:117] "RemoveContainer" containerID="b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68"
Sep 10 17:31:51.759123 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.759106    2568 scope.go:117] "RemoveContainer" containerID="b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68"
Sep 10 17:31:51.759502 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.759482    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68\": container with ID starting with b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68 not found: ID does not exist" containerID="b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68"
Sep 10 17:31:51.759571 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.759509    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68"} err="failed to get container status \"b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68\": rpc error: code = NotFound desc = could not find container \"b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68\": container with ID starting with b690c593eda0af8a9e72c504fd0721cb0e12205fac57cab0cc56829812866c68 not found: ID does not exist"
Sep 10 17:31:51.763928 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.763906    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b2b526-2204-4185-a032-846fd9132aa5" path="/var/lib/kubelet/pods/21b2b526-2204-4185-a032-846fd9132aa5/volumes"
Sep 10 17:31:51.764292 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.764219    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" path="/var/lib/kubelet/pods/44ad6dd7-3b1c-459b-b82e-3ffefa735b92/volumes"
Sep 10 17:31:51.770822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.770776    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2" podStartSLOduration=1.7707651800000002 podStartE2EDuration="1.77076518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:51.769957244 +0000 UTC m=+480.479551510" watchObservedRunningTime="2026-09-10 17:31:51.77076518 +0000 UTC m=+480.480359447"
Sep 10 17:31:51.772359 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:51.772337    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sp5pd\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="44ad6dd7-3b1c-459b-b82e-3ffefa735b92" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sp5pd"
Sep 10 17:31:51.803349 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:51.803294    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v" podStartSLOduration=1.803277897 podStartE2EDuration="1.803277897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:51.799581514 +0000 UTC m=+480.509175780" watchObservedRunningTime="2026-09-10 17:31:51.803277897 +0000 UTC m=+480.512872163"
Sep 10 17:31:52.323696 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.323659    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"]
Sep 10 17:31:52.328616 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.328600    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:52.348393 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.348360    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"]
Sep 10 17:31:52.399973 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.399938    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rt9sj\" (UID: \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:52.400124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.399999    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkhxg\" (UniqueName: \"kubernetes.io/projected/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-kube-api-access-dkhxg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rt9sj\" (UID: \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:52.501347 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.501310    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rt9sj\" (UID: \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:52.501520 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.501366    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkhxg\" (UniqueName: \"kubernetes.io/projected/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-kube-api-access-dkhxg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rt9sj\" (UID: \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:52.501777 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.501751    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rt9sj\" (UID: \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:52.510699 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.510672    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkhxg\" (UniqueName: \"kubernetes.io/projected/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-kube-api-access-dkhxg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rt9sj\" (UID: \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:52.638355 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.638229    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:52.697526 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.697496    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6qqc6"
Sep 10 17:31:52.917565 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:52.917541    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"]
Sep 10 17:31:52.920048 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:31:52.920019    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccc5e0d_ad5b_4de7_b596_acb3e79809dc.slice/crio-64a9141bd0af7b9f88746a240b448b61826c6d1c114632f8d86c2b818391bf98 WatchSource:0}: Error finding container 64a9141bd0af7b9f88746a240b448b61826c6d1c114632f8d86c2b818391bf98: Status 404 returned error can't find the container with id 64a9141bd0af7b9f88746a240b448b61826c6d1c114632f8d86c2b818391bf98
Sep 10 17:31:53.763807 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:53.763774    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:31:53.764044 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:53.763821    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj" event={"ID":"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc","Type":"ContainerStarted","Data":"6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd"}
Sep 10 17:31:53.764044 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:53.763840    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj" event={"ID":"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc","Type":"ContainerStarted","Data":"64a9141bd0af7b9f88746a240b448b61826c6d1c114632f8d86c2b818391bf98"}
Sep 10 17:31:53.787745 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:53.787670    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj" podStartSLOduration=1.787647227 podStartE2EDuration="1.787647227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:53.785793772 +0000 UTC m=+482.495388033" watchObservedRunningTime="2026-09-10 17:31:53.787647227 +0000 UTC m=+482.497241494"
Sep 10 17:31:57.730430 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:57.730389    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78f7ccd44-5c57j" podUID="3d46d4c8-589e-49c3-a1b8-43be173b3f75" containerName="console" containerID="cri-o://e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e" gracePeriod=15
Sep 10 17:31:57.976732 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:57.976703    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:31:58.046837 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.046799    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-config\") pod \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") "
Sep 10 17:31:58.047020 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.046844    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-service-ca\") pod \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") "
Sep 10 17:31:58.047020 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.046895    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-oauth-config\") pod \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") "
Sep 10 17:31:58.047020 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.046936    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7nxs\" (UniqueName: \"kubernetes.io/projected/3d46d4c8-589e-49c3-a1b8-43be173b3f75-kube-api-access-v7nxs\") pod \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") "
Sep 10 17:31:58.047020 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.046960    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-oauth-serving-cert\") pod \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") "
Sep 10 17:31:58.047020 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.047021    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-trusted-ca-bundle\") pod \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") "
Sep 10 17:31:58.047319 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.047049    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-serving-cert\") pod \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\" (UID: \"3d46d4c8-589e-49c3-a1b8-43be173b3f75\") "
Sep 10 17:31:58.047403 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.047362    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-service-ca" (OuterVolumeSpecName: "service-ca") pod "3d46d4c8-589e-49c3-a1b8-43be173b3f75" (UID: "3d46d4c8-589e-49c3-a1b8-43be173b3f75"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:31:58.047403 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.047366    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-config" (OuterVolumeSpecName: "console-config") pod "3d46d4c8-589e-49c3-a1b8-43be173b3f75" (UID: "3d46d4c8-589e-49c3-a1b8-43be173b3f75"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:31:58.047577 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.047461    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3d46d4c8-589e-49c3-a1b8-43be173b3f75" (UID: "3d46d4c8-589e-49c3-a1b8-43be173b3f75"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:31:58.047577 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.047489    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3d46d4c8-589e-49c3-a1b8-43be173b3f75" (UID: "3d46d4c8-589e-49c3-a1b8-43be173b3f75"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:31:58.049184 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.049160    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3d46d4c8-589e-49c3-a1b8-43be173b3f75" (UID: "3d46d4c8-589e-49c3-a1b8-43be173b3f75"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:31:58.049285 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.049177    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3d46d4c8-589e-49c3-a1b8-43be173b3f75" (UID: "3d46d4c8-589e-49c3-a1b8-43be173b3f75"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:31:58.049342 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.049294    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d46d4c8-589e-49c3-a1b8-43be173b3f75-kube-api-access-v7nxs" (OuterVolumeSpecName: "kube-api-access-v7nxs") pod "3d46d4c8-589e-49c3-a1b8-43be173b3f75" (UID: "3d46d4c8-589e-49c3-a1b8-43be173b3f75"). InnerVolumeSpecName "kube-api-access-v7nxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:31:58.148169 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.148137    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-oauth-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:58.148169 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.148165    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nxs\" (UniqueName: \"kubernetes.io/projected/3d46d4c8-589e-49c3-a1b8-43be173b3f75-kube-api-access-v7nxs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:58.148169 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.148178    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-oauth-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:58.148433 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.148189    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-trusted-ca-bundle\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:58.148433 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.148202    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-serving-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:58.148433 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.148213    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-console-config\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:58.148433 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.148225    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d46d4c8-589e-49c3-a1b8-43be173b3f75-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:31:58.785824 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.785786    2568 generic.go:356] "Generic (PLEG): container finished" podID="3d46d4c8-589e-49c3-a1b8-43be173b3f75" containerID="e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e" exitCode=0
Sep 10 17:31:58.786323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.785842    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f7ccd44-5c57j"
Sep 10 17:31:58.786323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.785873    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f7ccd44-5c57j" event={"ID":"3d46d4c8-589e-49c3-a1b8-43be173b3f75","Type":"ContainerDied","Data":"e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e"}
Sep 10 17:31:58.786323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.785937    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f7ccd44-5c57j" event={"ID":"3d46d4c8-589e-49c3-a1b8-43be173b3f75","Type":"ContainerDied","Data":"fe49ead90623962040409332cd1dbcc3536ba465e9328d65ea233cd037a179c1"}
Sep 10 17:31:58.786323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.785969    2568 scope.go:117] "RemoveContainer" containerID="e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e"
Sep 10 17:31:58.797381 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.797358    2568 scope.go:117] "RemoveContainer" containerID="e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e"
Sep 10 17:31:58.797697 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:31:58.797676    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e\": container with ID starting with e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e not found: ID does not exist" containerID="e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e"
Sep 10 17:31:58.797741 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.797707    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e"} err="failed to get container status \"e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e\": rpc error: code = NotFound desc = could not find container \"e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e\": container with ID starting with e988f3e1d231bfa4a14b6f852dc8c6a85009271413a6cf130daab9d8deafab1e not found: ID does not exist"
Sep 10 17:31:58.809200 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.809174    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78f7ccd44-5c57j"]
Sep 10 17:31:58.814061 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:58.814042    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78f7ccd44-5c57j"]
Sep 10 17:31:59.762851 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:31:59.762816    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d46d4c8-589e-49c3-a1b8-43be173b3f75" path="/var/lib/kubelet/pods/3d46d4c8-589e-49c3-a1b8-43be173b3f75/volumes"
Sep 10 17:32:02.754935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:02.754905    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:32:02.755328 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:02.755094    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mtmz2"
Sep 10 17:32:04.769040 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:04.769005    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:32:04.830010 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:04.829980    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"]
Sep 10 17:32:04.830197 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:04.830175    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v" podUID="a6c3e358-9880-4d4a-af39-5752131d94b2" containerName="manager" containerID="cri-o://d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2" gracePeriod=10
Sep 10 17:32:05.066898 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.066872    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:32:05.208763 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.208734    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2bpp\" (UniqueName: \"kubernetes.io/projected/a6c3e358-9880-4d4a-af39-5752131d94b2-kube-api-access-f2bpp\") pod \"a6c3e358-9880-4d4a-af39-5752131d94b2\" (UID: \"a6c3e358-9880-4d4a-af39-5752131d94b2\") "
Sep 10 17:32:05.208923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.208811    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6c3e358-9880-4d4a-af39-5752131d94b2-extensions-socket-volume\") pod \"a6c3e358-9880-4d4a-af39-5752131d94b2\" (UID: \"a6c3e358-9880-4d4a-af39-5752131d94b2\") "
Sep 10 17:32:05.209207 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.209177    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c3e358-9880-4d4a-af39-5752131d94b2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a6c3e358-9880-4d4a-af39-5752131d94b2" (UID: "a6c3e358-9880-4d4a-af39-5752131d94b2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:32:05.210647 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.210625    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c3e358-9880-4d4a-af39-5752131d94b2-kube-api-access-f2bpp" (OuterVolumeSpecName: "kube-api-access-f2bpp") pod "a6c3e358-9880-4d4a-af39-5752131d94b2" (UID: "a6c3e358-9880-4d4a-af39-5752131d94b2"). InnerVolumeSpecName "kube-api-access-f2bpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:05.309732 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.309652    2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6c3e358-9880-4d4a-af39-5752131d94b2-extensions-socket-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:32:05.309732 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.309679    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2bpp\" (UniqueName: \"kubernetes.io/projected/a6c3e358-9880-4d4a-af39-5752131d94b2-kube-api-access-f2bpp\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:32:05.821357 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.821324    2568 generic.go:356] "Generic (PLEG): container finished" podID="a6c3e358-9880-4d4a-af39-5752131d94b2" containerID="d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2" exitCode=0
Sep 10 17:32:05.821759 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.821364    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"
Sep 10 17:32:05.821759 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.821397    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v" event={"ID":"a6c3e358-9880-4d4a-af39-5752131d94b2","Type":"ContainerDied","Data":"d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2"}
Sep 10 17:32:05.821759 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.821439    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v" event={"ID":"a6c3e358-9880-4d4a-af39-5752131d94b2","Type":"ContainerDied","Data":"1896f076d1228ee3c498c836fd0601680fde8c22a1433798d3ba8dc898e08360"}
Sep 10 17:32:05.821759 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.821460    2568 scope.go:117] "RemoveContainer" containerID="d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2"
Sep 10 17:32:05.829477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.829458    2568 scope.go:117] "RemoveContainer" containerID="d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2"
Sep 10 17:32:05.829770 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:32:05.829748    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2\": container with ID starting with d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2 not found: ID does not exist" containerID="d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2"
Sep 10 17:32:05.829863 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.829776    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2"} err="failed to get container status \"d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2\": rpc error: code = NotFound desc = could not find container \"d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2\": container with ID starting with d5a84bfc599c2449ccfcbff16bfb3c08cbdf658e5649a3cc45d6bcc62a1289a2 not found: ID does not exist"
Sep 10 17:32:05.847540 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.847506    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"]
Sep 10 17:32:05.853493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:05.853460    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7pb5v"]
Sep 10 17:32:07.763294 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:07.763260    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c3e358-9880-4d4a-af39-5752131d94b2" path="/var/lib/kubelet/pods/a6c3e358-9880-4d4a-af39-5752131d94b2/volumes"
Sep 10 17:32:23.651820 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.651780    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k89m2"]
Sep 10 17:32:23.652226 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.652098    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c3e358-9880-4d4a-af39-5752131d94b2" containerName="manager"
Sep 10 17:32:23.652226 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.652109    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c3e358-9880-4d4a-af39-5752131d94b2" containerName="manager"
Sep 10 17:32:23.652226 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.652121    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d46d4c8-589e-49c3-a1b8-43be173b3f75" containerName="console"
Sep 10 17:32:23.652226 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.652126    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d46d4c8-589e-49c3-a1b8-43be173b3f75" containerName="console"
Sep 10 17:32:23.652226 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.652180    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c3e358-9880-4d4a-af39-5752131d94b2" containerName="manager"
Sep 10 17:32:23.652226 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.652190    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d46d4c8-589e-49c3-a1b8-43be173b3f75" containerName="console"
Sep 10 17:32:23.654147 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.654131    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:23.656915 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.656893    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8jzgf\""
Sep 10 17:32:23.656915 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.656912    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 17:32:23.664343 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.664318    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k89m2"]
Sep 10 17:32:23.748754 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.748710    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k89m2"]
Sep 10 17:32:23.761663 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.761626    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/16f6e69b-3333-45fa-95a8-f95cc833ceb8-config-file\") pod \"limitador-limitador-8588749977-k89m2\" (UID: \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\") " pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:23.761789 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.761684    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfbsc\" (UniqueName: \"kubernetes.io/projected/16f6e69b-3333-45fa-95a8-f95cc833ceb8-kube-api-access-gfbsc\") pod \"limitador-limitador-8588749977-k89m2\" (UID: \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\") " pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:23.862692 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.862657    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfbsc\" (UniqueName: \"kubernetes.io/projected/16f6e69b-3333-45fa-95a8-f95cc833ceb8-kube-api-access-gfbsc\") pod \"limitador-limitador-8588749977-k89m2\" (UID: \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\") " pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:23.862859 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.862723    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/16f6e69b-3333-45fa-95a8-f95cc833ceb8-config-file\") pod \"limitador-limitador-8588749977-k89m2\" (UID: \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\") " pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:23.863387 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.863370    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/16f6e69b-3333-45fa-95a8-f95cc833ceb8-config-file\") pod \"limitador-limitador-8588749977-k89m2\" (UID: \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\") " pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:23.873207 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.873181    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfbsc\" (UniqueName: \"kubernetes.io/projected/16f6e69b-3333-45fa-95a8-f95cc833ceb8-kube-api-access-gfbsc\") pod \"limitador-limitador-8588749977-k89m2\" (UID: \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\") " pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:23.964662 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:23.964560    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:24.239003 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.238976    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k89m2"]
Sep 10 17:32:24.470342 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.470308    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65p2w"]
Sep 10 17:32:24.473485 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.473466    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w"
Sep 10 17:32:24.476635 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.476617    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q6dps\""
Sep 10 17:32:24.481559 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.481540    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65p2w"]
Sep 10 17:32:24.668263 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.668211    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cdw\" (UniqueName: \"kubernetes.io/projected/698cd0a4-5389-492b-a668-81cc81de7b66-kube-api-access-j2cdw\") pod \"authorino-7bcfd8f9cd-65p2w\" (UID: \"698cd0a4-5389-492b-a668-81cc81de7b66\") " pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w"
Sep 10 17:32:24.769486 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.769446    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cdw\" (UniqueName: \"kubernetes.io/projected/698cd0a4-5389-492b-a668-81cc81de7b66-kube-api-access-j2cdw\") pod \"authorino-7bcfd8f9cd-65p2w\" (UID: \"698cd0a4-5389-492b-a668-81cc81de7b66\") " pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w"
Sep 10 17:32:24.778190 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.778167    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cdw\" (UniqueName: \"kubernetes.io/projected/698cd0a4-5389-492b-a668-81cc81de7b66-kube-api-access-j2cdw\") pod \"authorino-7bcfd8f9cd-65p2w\" (UID: \"698cd0a4-5389-492b-a668-81cc81de7b66\") " pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w"
Sep 10 17:32:24.782894 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.782874    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w"
Sep 10 17:32:24.916886 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:24.916849    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k89m2" event={"ID":"16f6e69b-3333-45fa-95a8-f95cc833ceb8","Type":"ContainerStarted","Data":"c72611588afe334ea27e1b28f086005fe24ce2341ceee1eb041113344490f375"}
Sep 10 17:32:25.078539 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:25.078321    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65p2w"]
Sep 10 17:32:25.080874 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:32:25.080842    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod698cd0a4_5389_492b_a668_81cc81de7b66.slice/crio-4309c782451269d0bad4a854b9f727957d8d82c89c2488fa052e2dc1886d876c WatchSource:0}: Error finding container 4309c782451269d0bad4a854b9f727957d8d82c89c2488fa052e2dc1886d876c: Status 404 returned error can't find the container with id 4309c782451269d0bad4a854b9f727957d8d82c89c2488fa052e2dc1886d876c
Sep 10 17:32:25.924696 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:25.924659    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w" event={"ID":"698cd0a4-5389-492b-a668-81cc81de7b66","Type":"ContainerStarted","Data":"4309c782451269d0bad4a854b9f727957d8d82c89c2488fa052e2dc1886d876c"}
Sep 10 17:32:25.926510 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:25.926478    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k89m2" event={"ID":"16f6e69b-3333-45fa-95a8-f95cc833ceb8","Type":"ContainerStarted","Data":"1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5"}
Sep 10 17:32:25.926646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:25.926590    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:25.948380 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:25.948152    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-k89m2" podStartSLOduration=1.718940526 podStartE2EDuration="2.948138061s" totalImagesPullingTime="1.229197535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:25.946169996 +0000 UTC m=+514.655764265" watchObservedRunningTime="2026-09-10 17:32:25.948138061 +0000 UTC m=+514.657732326"
Sep 10 17:32:26.933047 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:26.932999    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w" event={"ID":"698cd0a4-5389-492b-a668-81cc81de7b66","Type":"ContainerStarted","Data":"f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8"}
Sep 10 17:32:26.948456 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:26.948412    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w" podStartSLOduration=1.28794615 podStartE2EDuration="2.948385671s" totalImagesPullingTime="1.660439521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:26.948139329 +0000 UTC m=+515.657733596" watchObservedRunningTime="2026-09-10 17:32:26.948385671 +0000 UTC m=+515.657979939"
Sep 10 17:32:28.342049 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:28.342010    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65p2w"]
Sep 10 17:32:28.942398 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:28.942339    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w" podUID="698cd0a4-5389-492b-a668-81cc81de7b66" containerName="authorino" containerID="cri-o://f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8" gracePeriod=30
Sep 10 17:32:29.180858 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.180832    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w"
Sep 10 17:32:29.202024 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.201948    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2cdw\" (UniqueName: \"kubernetes.io/projected/698cd0a4-5389-492b-a668-81cc81de7b66-kube-api-access-j2cdw\") pod \"698cd0a4-5389-492b-a668-81cc81de7b66\" (UID: \"698cd0a4-5389-492b-a668-81cc81de7b66\") "
Sep 10 17:32:29.204044 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.204012    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698cd0a4-5389-492b-a668-81cc81de7b66-kube-api-access-j2cdw" (OuterVolumeSpecName: "kube-api-access-j2cdw") pod "698cd0a4-5389-492b-a668-81cc81de7b66" (UID: "698cd0a4-5389-492b-a668-81cc81de7b66"). InnerVolumeSpecName "kube-api-access-j2cdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:29.303007 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.302975    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2cdw\" (UniqueName: \"kubernetes.io/projected/698cd0a4-5389-492b-a668-81cc81de7b66-kube-api-access-j2cdw\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:32:29.947539 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.947499    2568 generic.go:356] "Generic (PLEG): container finished" podID="698cd0a4-5389-492b-a668-81cc81de7b66" containerID="f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8" exitCode=0
Sep 10 17:32:29.947539 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.947537    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w"
Sep 10 17:32:29.948060 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.947545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w" event={"ID":"698cd0a4-5389-492b-a668-81cc81de7b66","Type":"ContainerDied","Data":"f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8"}
Sep 10 17:32:29.948060 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.947588    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-65p2w" event={"ID":"698cd0a4-5389-492b-a668-81cc81de7b66","Type":"ContainerDied","Data":"4309c782451269d0bad4a854b9f727957d8d82c89c2488fa052e2dc1886d876c"}
Sep 10 17:32:29.948060 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.947604    2568 scope.go:117] "RemoveContainer" containerID="f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8"
Sep 10 17:32:29.955853 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.955834    2568 scope.go:117] "RemoveContainer" containerID="f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8"
Sep 10 17:32:29.956176 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:32:29.956156    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8\": container with ID starting with f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8 not found: ID does not exist" containerID="f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8"
Sep 10 17:32:29.956247 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.956185    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8"} err="failed to get container status \"f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8\": rpc error: code = NotFound desc = could not find container \"f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8\": container with ID starting with f3f4d67b64580e29b33b8da44007ddab8c8f89dc0d20710f1194d2f19b7c6be8 not found: ID does not exist"
Sep 10 17:32:29.966779 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.966749    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65p2w"]
Sep 10 17:32:29.970534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:29.970514    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-65p2w"]
Sep 10 17:32:31.762408 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:31.762369    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="698cd0a4-5389-492b-a668-81cc81de7b66" path="/var/lib/kubelet/pods/698cd0a4-5389-492b-a668-81cc81de7b66/volumes"
Sep 10 17:32:36.934543 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:36.934513    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:39.345182 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.345152    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k89m2"]
Sep 10 17:32:39.345591 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.345368    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-k89m2" podUID="16f6e69b-3333-45fa-95a8-f95cc833ceb8" containerName="limitador" containerID="cri-o://1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5" gracePeriod=30
Sep 10 17:32:39.885640 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.885613    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:39.979136 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.979053    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfbsc\" (UniqueName: \"kubernetes.io/projected/16f6e69b-3333-45fa-95a8-f95cc833ceb8-kube-api-access-gfbsc\") pod \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\" (UID: \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\") "
Sep 10 17:32:39.979136 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.979111    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/16f6e69b-3333-45fa-95a8-f95cc833ceb8-config-file\") pod \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\" (UID: \"16f6e69b-3333-45fa-95a8-f95cc833ceb8\") "
Sep 10 17:32:39.979511 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.979485    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f6e69b-3333-45fa-95a8-f95cc833ceb8-config-file" (OuterVolumeSpecName: "config-file") pod "16f6e69b-3333-45fa-95a8-f95cc833ceb8" (UID: "16f6e69b-3333-45fa-95a8-f95cc833ceb8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:32:39.981247 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.981213    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f6e69b-3333-45fa-95a8-f95cc833ceb8-kube-api-access-gfbsc" (OuterVolumeSpecName: "kube-api-access-gfbsc") pod "16f6e69b-3333-45fa-95a8-f95cc833ceb8" (UID: "16f6e69b-3333-45fa-95a8-f95cc833ceb8"). InnerVolumeSpecName "kube-api-access-gfbsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:39.998610 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.998575    2568 generic.go:356] "Generic (PLEG): container finished" podID="16f6e69b-3333-45fa-95a8-f95cc833ceb8" containerID="1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5" exitCode=0
Sep 10 17:32:39.998610 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.998610    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k89m2"
Sep 10 17:32:39.998777 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.998623    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k89m2" event={"ID":"16f6e69b-3333-45fa-95a8-f95cc833ceb8","Type":"ContainerDied","Data":"1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5"}
Sep 10 17:32:39.998777 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.998680    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k89m2" event={"ID":"16f6e69b-3333-45fa-95a8-f95cc833ceb8","Type":"ContainerDied","Data":"c72611588afe334ea27e1b28f086005fe24ce2341ceee1eb041113344490f375"}
Sep 10 17:32:39.998777 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:39.998710    2568 scope.go:117] "RemoveContainer" containerID="1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5"
Sep 10 17:32:40.009185 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:40.009162    2568 scope.go:117] "RemoveContainer" containerID="1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5"
Sep 10 17:32:40.009582 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:32:40.009558    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5\": container with ID starting with 1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5 not found: ID does not exist" containerID="1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5"
Sep 10 17:32:40.009688 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:40.009593    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5"} err="failed to get container status \"1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5\": rpc error: code = NotFound desc = could not find container \"1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5\": container with ID starting with 1991613831dd1b1f19fb34a7908c907a8abbd5ee7c1384648673e4c29d2af5f5 not found: ID does not exist"
Sep 10 17:32:40.022031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:40.022006    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k89m2"]
Sep 10 17:32:40.023924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:40.023899    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k89m2"]
Sep 10 17:32:40.079557 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:40.079525    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfbsc\" (UniqueName: \"kubernetes.io/projected/16f6e69b-3333-45fa-95a8-f95cc833ceb8-kube-api-access-gfbsc\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:32:40.079557 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:40.079554    2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/16f6e69b-3333-45fa-95a8-f95cc833ceb8-config-file\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:32:41.763540 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:41.763501    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16f6e69b-3333-45fa-95a8-f95cc833ceb8" path="/var/lib/kubelet/pods/16f6e69b-3333-45fa-95a8-f95cc833ceb8/volumes"
Sep 10 17:32:45.309488 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.309410    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"]
Sep 10 17:32:45.309839 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.309773    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16f6e69b-3333-45fa-95a8-f95cc833ceb8" containerName="limitador"
Sep 10 17:32:45.309839 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.309786    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="16f6e69b-3333-45fa-95a8-f95cc833ceb8" containerName="limitador"
Sep 10 17:32:45.309839 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.309798    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698cd0a4-5389-492b-a668-81cc81de7b66" containerName="authorino"
Sep 10 17:32:45.309839 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.309803    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="698cd0a4-5389-492b-a668-81cc81de7b66" containerName="authorino"
Sep 10 17:32:45.309987 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.309852    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="16f6e69b-3333-45fa-95a8-f95cc833ceb8" containerName="limitador"
Sep 10 17:32:45.309987 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.309863    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="698cd0a4-5389-492b-a668-81cc81de7b66" containerName="authorino"
Sep 10 17:32:45.315701 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.315681    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:45.320215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.320195    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 17:32:45.321060 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.320982    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7xpqv\""
Sep 10 17:32:45.321166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.321009    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 17:32:45.321364 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.321346    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:32:45.322892 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.322871    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"]
Sep 10 17:32:45.422123 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.422090    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q6tb\" (UniqueName: \"kubernetes.io/projected/7f00e367-c983-489a-a34c-f02cd6957354-kube-api-access-8q6tb\") pod \"postgres-69fb4594cf-5d2zw\" (UID: \"7f00e367-c983-489a-a34c-f02cd6957354\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:45.422336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.422138    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f00e367-c983-489a-a34c-f02cd6957354-data\") pod \"postgres-69fb4594cf-5d2zw\" (UID: \"7f00e367-c983-489a-a34c-f02cd6957354\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:45.523406 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.523375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6tb\" (UniqueName: \"kubernetes.io/projected/7f00e367-c983-489a-a34c-f02cd6957354-kube-api-access-8q6tb\") pod \"postgres-69fb4594cf-5d2zw\" (UID: \"7f00e367-c983-489a-a34c-f02cd6957354\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:45.523555 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.523411    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f00e367-c983-489a-a34c-f02cd6957354-data\") pod \"postgres-69fb4594cf-5d2zw\" (UID: \"7f00e367-c983-489a-a34c-f02cd6957354\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:45.523780 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.523761    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f00e367-c983-489a-a34c-f02cd6957354-data\") pod \"postgres-69fb4594cf-5d2zw\" (UID: \"7f00e367-c983-489a-a34c-f02cd6957354\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:45.534139 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.534118    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6tb\" (UniqueName: \"kubernetes.io/projected/7f00e367-c983-489a-a34c-f02cd6957354-kube-api-access-8q6tb\") pod \"postgres-69fb4594cf-5d2zw\" (UID: \"7f00e367-c983-489a-a34c-f02cd6957354\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:45.625656 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.625564    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:45.895474 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:45.895392    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"]
Sep 10 17:32:45.898653 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:32:45.898624    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f00e367_c983_489a_a34c_f02cd6957354.slice/crio-46e02621dbe61f704558f9ba8b779e53acd61a94278c9cc5973d3275332f9c18 WatchSource:0}: Error finding container 46e02621dbe61f704558f9ba8b779e53acd61a94278c9cc5973d3275332f9c18: Status 404 returned error can't find the container with id 46e02621dbe61f704558f9ba8b779e53acd61a94278c9cc5973d3275332f9c18
Sep 10 17:32:46.027813 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:46.027776    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw" event={"ID":"7f00e367-c983-489a-a34c-f02cd6957354","Type":"ContainerStarted","Data":"46e02621dbe61f704558f9ba8b779e53acd61a94278c9cc5973d3275332f9c18"}
Sep 10 17:32:51.057905 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:51.057873    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw" event={"ID":"7f00e367-c983-489a-a34c-f02cd6957354","Type":"ContainerStarted","Data":"f0913820e230e422de5c1f4b5fc1e3bc08ecabf40c40347718220e059e1158b7"}
Sep 10 17:32:51.058276 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:51.057936    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:51.077224 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:51.077169    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw" podStartSLOduration=1.110134401 podStartE2EDuration="6.077152481s" totalImagesPullingTime="4.96701808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:51.076188844 +0000 UTC m=+539.785783110" watchObservedRunningTime="2026-09-10 17:32:51.077152481 +0000 UTC m=+539.786746746"
Sep 10 17:32:57.090944 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.090916    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5d2zw"
Sep 10 17:32:57.641521 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.641481    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-j5fdq"]
Sep 10 17:32:57.645072 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.645052    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-j5fdq"
Sep 10 17:32:57.648440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.648420    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q6dps\""
Sep 10 17:32:57.653208 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.653186    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-j5fdq"]
Sep 10 17:32:57.691358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.691323    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-j5fdq"]
Sep 10 17:32:57.691566 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:32:57.691548    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hqpt7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-j5fdq" podUID="71af2233-8eb6-4004-87e9-337e3f8255f2"
Sep 10 17:32:57.726891 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.726855    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqpt7\" (UniqueName: \"kubernetes.io/projected/71af2233-8eb6-4004-87e9-337e3f8255f2-kube-api-access-hqpt7\") pod \"authorino-557f876f69-j5fdq\" (UID: \"71af2233-8eb6-4004-87e9-337e3f8255f2\") " pod="kuadrant-system/authorino-557f876f69-j5fdq"
Sep 10 17:32:57.727319 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.727297    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jcsfk"]
Sep 10 17:32:57.730723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.730697    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:57.733342 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.733323    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 17:32:57.740063 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.740037    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jcsfk"]
Sep 10 17:32:57.827897 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.827860    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpt7\" (UniqueName: \"kubernetes.io/projected/71af2233-8eb6-4004-87e9-337e3f8255f2-kube-api-access-hqpt7\") pod \"authorino-557f876f69-j5fdq\" (UID: \"71af2233-8eb6-4004-87e9-337e3f8255f2\") " pod="kuadrant-system/authorino-557f876f69-j5fdq"
Sep 10 17:32:57.828083 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.827935    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/220f287b-4120-46ce-902f-7a22e8730274-tls-cert\") pod \"authorino-6d4dfc649f-jcsfk\" (UID: \"220f287b-4120-46ce-902f-7a22e8730274\") " pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:57.828083 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.827969    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fwq\" (UniqueName: \"kubernetes.io/projected/220f287b-4120-46ce-902f-7a22e8730274-kube-api-access-s7fwq\") pod \"authorino-6d4dfc649f-jcsfk\" (UID: \"220f287b-4120-46ce-902f-7a22e8730274\") " pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:57.835682 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.835654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpt7\" (UniqueName: \"kubernetes.io/projected/71af2233-8eb6-4004-87e9-337e3f8255f2-kube-api-access-hqpt7\") pod \"authorino-557f876f69-j5fdq\" (UID: \"71af2233-8eb6-4004-87e9-337e3f8255f2\") " pod="kuadrant-system/authorino-557f876f69-j5fdq"
Sep 10 17:32:57.875259 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.875200    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jcsfk"]
Sep 10 17:32:57.875488 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:32:57.875469    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s7fwq tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-jcsfk" podUID="220f287b-4120-46ce-902f-7a22e8730274"
Sep 10 17:32:57.904516 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.904438    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-645c68ff47-gnkzj"]
Sep 10 17:32:57.907750 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.907730    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:32:57.917370 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.917346    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-645c68ff47-gnkzj"]
Sep 10 17:32:57.929254 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.929208    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/220f287b-4120-46ce-902f-7a22e8730274-tls-cert\") pod \"authorino-6d4dfc649f-jcsfk\" (UID: \"220f287b-4120-46ce-902f-7a22e8730274\") " pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:57.929375 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.929256    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fwq\" (UniqueName: \"kubernetes.io/projected/220f287b-4120-46ce-902f-7a22e8730274-kube-api-access-s7fwq\") pod \"authorino-6d4dfc649f-jcsfk\" (UID: \"220f287b-4120-46ce-902f-7a22e8730274\") " pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:57.932128 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.932105    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/220f287b-4120-46ce-902f-7a22e8730274-tls-cert\") pod \"authorino-6d4dfc649f-jcsfk\" (UID: \"220f287b-4120-46ce-902f-7a22e8730274\") " pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:57.937400 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:57.937372    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fwq\" (UniqueName: \"kubernetes.io/projected/220f287b-4120-46ce-902f-7a22e8730274-kube-api-access-s7fwq\") pod \"authorino-6d4dfc649f-jcsfk\" (UID: \"220f287b-4120-46ce-902f-7a22e8730274\") " pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:58.029887 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.029844    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6676e15-3321-4ff4-bf33-349abf9b7727-tls-cert\") pod \"authorino-645c68ff47-gnkzj\" (UID: \"b6676e15-3321-4ff4-bf33-349abf9b7727\") " pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:32:58.029887 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.029889    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9zn\" (UniqueName: \"kubernetes.io/projected/b6676e15-3321-4ff4-bf33-349abf9b7727-kube-api-access-sq9zn\") pod \"authorino-645c68ff47-gnkzj\" (UID: \"b6676e15-3321-4ff4-bf33-349abf9b7727\") " pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:32:58.087971 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.087942    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:58.088137 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.087942    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-j5fdq"
Sep 10 17:32:58.095194 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.095172    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:58.097888 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.097870    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-j5fdq"
Sep 10 17:32:58.130554 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.130522    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6676e15-3321-4ff4-bf33-349abf9b7727-tls-cert\") pod \"authorino-645c68ff47-gnkzj\" (UID: \"b6676e15-3321-4ff4-bf33-349abf9b7727\") " pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:32:58.130554 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.130565    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9zn\" (UniqueName: \"kubernetes.io/projected/b6676e15-3321-4ff4-bf33-349abf9b7727-kube-api-access-sq9zn\") pod \"authorino-645c68ff47-gnkzj\" (UID: \"b6676e15-3321-4ff4-bf33-349abf9b7727\") " pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:32:58.133176 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.133151    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6676e15-3321-4ff4-bf33-349abf9b7727-tls-cert\") pod \"authorino-645c68ff47-gnkzj\" (UID: \"b6676e15-3321-4ff4-bf33-349abf9b7727\") " pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:32:58.143082 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.143057    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9zn\" (UniqueName: \"kubernetes.io/projected/b6676e15-3321-4ff4-bf33-349abf9b7727-kube-api-access-sq9zn\") pod \"authorino-645c68ff47-gnkzj\" (UID: \"b6676e15-3321-4ff4-bf33-349abf9b7727\") " pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:32:58.218015 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.217933    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:32:58.231026 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.231003    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqpt7\" (UniqueName: \"kubernetes.io/projected/71af2233-8eb6-4004-87e9-337e3f8255f2-kube-api-access-hqpt7\") pod \"71af2233-8eb6-4004-87e9-337e3f8255f2\" (UID: \"71af2233-8eb6-4004-87e9-337e3f8255f2\") "
Sep 10 17:32:58.231127 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.231082    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/220f287b-4120-46ce-902f-7a22e8730274-tls-cert\") pod \"220f287b-4120-46ce-902f-7a22e8730274\" (UID: \"220f287b-4120-46ce-902f-7a22e8730274\") "
Sep 10 17:32:58.231193 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.231136    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7fwq\" (UniqueName: \"kubernetes.io/projected/220f287b-4120-46ce-902f-7a22e8730274-kube-api-access-s7fwq\") pod \"220f287b-4120-46ce-902f-7a22e8730274\" (UID: \"220f287b-4120-46ce-902f-7a22e8730274\") "
Sep 10 17:32:58.233108 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.233082    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220f287b-4120-46ce-902f-7a22e8730274-kube-api-access-s7fwq" (OuterVolumeSpecName: "kube-api-access-s7fwq") pod "220f287b-4120-46ce-902f-7a22e8730274" (UID: "220f287b-4120-46ce-902f-7a22e8730274"). InnerVolumeSpecName "kube-api-access-s7fwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:58.233194 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.233126    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71af2233-8eb6-4004-87e9-337e3f8255f2-kube-api-access-hqpt7" (OuterVolumeSpecName: "kube-api-access-hqpt7") pod "71af2233-8eb6-4004-87e9-337e3f8255f2" (UID: "71af2233-8eb6-4004-87e9-337e3f8255f2"). InnerVolumeSpecName "kube-api-access-hqpt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:58.233194 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.233126    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220f287b-4120-46ce-902f-7a22e8730274-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "220f287b-4120-46ce-902f-7a22e8730274" (UID: "220f287b-4120-46ce-902f-7a22e8730274"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:32:58.332583 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.332547    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fwq\" (UniqueName: \"kubernetes.io/projected/220f287b-4120-46ce-902f-7a22e8730274-kube-api-access-s7fwq\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:32:58.332583 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.332579    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqpt7\" (UniqueName: \"kubernetes.io/projected/71af2233-8eb6-4004-87e9-337e3f8255f2-kube-api-access-hqpt7\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:32:58.332583 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.332589    2568 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/220f287b-4120-46ce-902f-7a22e8730274-tls-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:32:58.484435 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:58.484326    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-645c68ff47-gnkzj"]
Sep 10 17:32:58.486474 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:32:58.486450    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6676e15_3321_4ff4_bf33_349abf9b7727.slice/crio-7ab892316af4d9554235d6469cfb5dc661c06bb6abac1d09a6e8c7706ff53b08 WatchSource:0}: Error finding container 7ab892316af4d9554235d6469cfb5dc661c06bb6abac1d09a6e8c7706ff53b08: Status 404 returned error can't find the container with id 7ab892316af4d9554235d6469cfb5dc661c06bb6abac1d09a6e8c7706ff53b08
Sep 10 17:32:59.095141 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.095044    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-j5fdq"
Sep 10 17:32:59.095141 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.095086    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-645c68ff47-gnkzj" event={"ID":"b6676e15-3321-4ff4-bf33-349abf9b7727","Type":"ContainerStarted","Data":"8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2"}
Sep 10 17:32:59.095141 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.095140    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-645c68ff47-gnkzj" event={"ID":"b6676e15-3321-4ff4-bf33-349abf9b7727","Type":"ContainerStarted","Data":"7ab892316af4d9554235d6469cfb5dc661c06bb6abac1d09a6e8c7706ff53b08"}
Sep 10 17:32:59.095742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.095260    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-jcsfk"
Sep 10 17:32:59.132568 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.132514    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-645c68ff47-gnkzj" podStartSLOduration=1.633414256 podStartE2EDuration="2.132496719s" totalImagesPullingTime="499.082463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:59.11125947 +0000 UTC m=+547.820853738" watchObservedRunningTime="2026-09-10 17:32:59.132496719 +0000 UTC m=+547.842090986"
Sep 10 17:32:59.144652 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.144616    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-j5fdq"]
Sep 10 17:32:59.151098 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.151068    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-j5fdq"]
Sep 10 17:32:59.179951 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.179921    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jcsfk"]
Sep 10 17:32:59.182856 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.182830    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jcsfk"]
Sep 10 17:32:59.764441 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.764407    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="220f287b-4120-46ce-902f-7a22e8730274" path="/var/lib/kubelet/pods/220f287b-4120-46ce-902f-7a22e8730274/volumes"
Sep 10 17:32:59.764644 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:32:59.764632    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71af2233-8eb6-4004-87e9-337e3f8255f2" path="/var/lib/kubelet/pods/71af2233-8eb6-4004-87e9-337e3f8255f2/volumes"
Sep 10 17:33:03.856434 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.856385    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fbd6964c8-xhkr4"]
Sep 10 17:33:03.859940 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.859917    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:03.866520 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.866205    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 17:33:03.866520 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.866307    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 17:33:03.866648 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.866602    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-spdc5\""
Sep 10 17:33:03.867458 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.867440    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 17:33:03.875709 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.875684    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fbd6964c8-xhkr4"]
Sep 10 17:33:03.982107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.982072    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:03.982107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.982117    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2bss\" (UniqueName: \"kubernetes.io/projected/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-kube-api-access-h2bss\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:03.982389 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:03.982157    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-metrics-certs\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:04.047072 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.047037    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-t7s8h"]
Sep 10 17:33:04.050423 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.050403    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.062770 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.062743    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-t7s8h"]
Sep 10 17:33:04.083420 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.083383    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:04.083584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.083428    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bss\" (UniqueName: \"kubernetes.io/projected/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-kube-api-access-h2bss\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:04.083584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.083473    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-metrics-certs\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:04.083584 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:33:04.083524    2568 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 10 17:33:04.083584 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:33:04.083581    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert podName:61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4 nodeName:}" failed. No retries permitted until 2026-09-10 17:33:04.583564912 +0000 UTC m=+553.293159155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert") pod "maas-controller-6fbd6964c8-xhkr4" (UID: "61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4") : secret "maas-controller-webhook-cert" not found
Sep 10 17:33:04.085882 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.085862    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-metrics-certs\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:04.092010 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.091988    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bss\" (UniqueName: \"kubernetes.io/projected/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-kube-api-access-h2bss\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:04.184335 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.184229    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbwl\" (UniqueName: \"kubernetes.io/projected/3a0c986c-7370-413b-b1ca-51ba670b1d65-kube-api-access-mmbwl\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.184335 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.184308    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-webhook-cert\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.184535 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.184361    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-metrics-certs\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.285442 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.285401    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-metrics-certs\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.285622 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.285472    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbwl\" (UniqueName: \"kubernetes.io/projected/3a0c986c-7370-413b-b1ca-51ba670b1d65-kube-api-access-mmbwl\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.285622 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.285511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-webhook-cert\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.287940 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.287921    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-webhook-cert\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.288065 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.288045    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-metrics-certs\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.293576 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.293557    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbwl\" (UniqueName: \"kubernetes.io/projected/3a0c986c-7370-413b-b1ca-51ba670b1d65-kube-api-access-mmbwl\") pod \"maas-controller-5c56cf8bbb-t7s8h\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") " pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.366567 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.366533    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:04.588102 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.588068    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:04.590430 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.590410    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert\") pod \"maas-controller-6fbd6964c8-xhkr4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") " pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:04.637912 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.637791    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-t7s8h"]
Sep 10 17:33:04.640642 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:33:04.640608    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0c986c_7370_413b_b1ca_51ba670b1d65.slice/crio-3e4eb653072412095cd262d121215021fc12fd8eef57bff458e8b2979f3fd1e1 WatchSource:0}: Error finding container 3e4eb653072412095cd262d121215021fc12fd8eef57bff458e8b2979f3fd1e1: Status 404 returned error can't find the container with id 3e4eb653072412095cd262d121215021fc12fd8eef57bff458e8b2979f3fd1e1
Sep 10 17:33:04.770592 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:04.770564    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:05.042024 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:05.041008    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fbd6964c8-xhkr4"]
Sep 10 17:33:05.043679 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:33:05.043653    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fa305b_a8c2_4adf_94ae_f0ebdd7a25a4.slice/crio-977f6d38b041728e60d4f797d165aae1f349fd889cc9486c57eb474f79049bf0 WatchSource:0}: Error finding container 977f6d38b041728e60d4f797d165aae1f349fd889cc9486c57eb474f79049bf0: Status 404 returned error can't find the container with id 977f6d38b041728e60d4f797d165aae1f349fd889cc9486c57eb474f79049bf0
Sep 10 17:33:05.122955 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:05.122918    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h" event={"ID":"3a0c986c-7370-413b-b1ca-51ba670b1d65","Type":"ContainerStarted","Data":"3e4eb653072412095cd262d121215021fc12fd8eef57bff458e8b2979f3fd1e1"}
Sep 10 17:33:05.124049 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:05.124022    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4" event={"ID":"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4","Type":"ContainerStarted","Data":"977f6d38b041728e60d4f797d165aae1f349fd889cc9486c57eb474f79049bf0"}
Sep 10 17:33:08.141754 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:08.141719    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h" event={"ID":"3a0c986c-7370-413b-b1ca-51ba670b1d65","Type":"ContainerStarted","Data":"7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c"}
Sep 10 17:33:08.142201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:08.141786    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:08.143694 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:08.143667    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4" event={"ID":"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4","Type":"ContainerStarted","Data":"8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5"}
Sep 10 17:33:08.143822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:08.143720    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:08.164164 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:08.164112    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h" podStartSLOduration=1.608382644 podStartE2EDuration="4.164096261s" totalImagesPullingTime="2.555713617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:08.162168364 +0000 UTC m=+556.871762631" watchObservedRunningTime="2026-09-10 17:33:08.164096261 +0000 UTC m=+556.873690594"
Sep 10 17:33:08.182552 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:08.182500    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4" podStartSLOduration=3.023427289 podStartE2EDuration="5.182481197s" totalImagesPullingTime="2.159053908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:08.181090379 +0000 UTC m=+556.890684660" watchObservedRunningTime="2026-09-10 17:33:08.182481197 +0000 UTC m=+556.892075463"
Sep 10 17:33:19.151408 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.151378    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:19.151821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.151444    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:33:19.224716 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.224688    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fbd6964c8-xhkr4"]
Sep 10 17:33:19.224895 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.224872    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4" podUID="61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" containerName="manager" containerID="cri-o://8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5" gracePeriod=30
Sep 10 17:33:19.463045 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.463021    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:19.619560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.619521    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert\") pod \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") "
Sep 10 17:33:19.619767 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.619628    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-metrics-certs\") pod \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") "
Sep 10 17:33:19.619767 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.619664    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2bss\" (UniqueName: \"kubernetes.io/projected/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-kube-api-access-h2bss\") pod \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\" (UID: \"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4\") "
Sep 10 17:33:19.621891 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.621845    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-kube-api-access-h2bss" (OuterVolumeSpecName: "kube-api-access-h2bss") pod "61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" (UID: "61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4"). InnerVolumeSpecName "kube-api-access-h2bss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:19.621891 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.621868    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" (UID: "61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:19.621891 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.621851    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" (UID: "61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:19.720643 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.720543    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:33:19.720643 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.720590    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2bss\" (UniqueName: \"kubernetes.io/projected/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-kube-api-access-h2bss\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:33:19.720643 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:19.720603    2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4-webhook-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:33:20.198843 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.198801    2568 generic.go:356] "Generic (PLEG): container finished" podID="61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" containerID="8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5" exitCode=0
Sep 10 17:33:20.199258 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.198858    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4"
Sep 10 17:33:20.199258 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.198883    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4" event={"ID":"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4","Type":"ContainerDied","Data":"8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5"}
Sep 10 17:33:20.199258 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.198940    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbd6964c8-xhkr4" event={"ID":"61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4","Type":"ContainerDied","Data":"977f6d38b041728e60d4f797d165aae1f349fd889cc9486c57eb474f79049bf0"}
Sep 10 17:33:20.199258 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.198962    2568 scope.go:117] "RemoveContainer" containerID="8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5"
Sep 10 17:33:20.207201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.207184    2568 scope.go:117] "RemoveContainer" containerID="8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5"
Sep 10 17:33:20.207485 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:33:20.207467    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5\": container with ID starting with 8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5 not found: ID does not exist" containerID="8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5"
Sep 10 17:33:20.207550 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.207501    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5"} err="failed to get container status \"8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5\": rpc error: code = NotFound desc = could not find container \"8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5\": container with ID starting with 8a917ebbfa9757a7e21b0cf90c00a1f7bb92cf1d58d3860d53f8bace5c2e94e5 not found: ID does not exist"
Sep 10 17:33:20.216624 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.216598    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fbd6964c8-xhkr4"]
Sep 10 17:33:20.222256 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:20.222221    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fbd6964c8-xhkr4"]
Sep 10 17:33:21.767837 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:21.767803    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" path="/var/lib/kubelet/pods/61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4/volumes"
Sep 10 17:33:37.900640 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.900609    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"]
Sep 10 17:33:37.901046 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.900930    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" containerName="manager"
Sep 10 17:33:37.901046 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.900940    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" containerName="manager"
Sep 10 17:33:37.901046 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.901007    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="61fa305b-a8c2-4adf-94ae-f0ebdd7a25a4" containerName="manager"
Sep 10 17:33:37.919489 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.919462    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"]
Sep 10 17:33:37.919668 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.919573    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:37.922617 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.922590    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 17:33:37.923081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.923056    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 17:33:37.923185 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.923060    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x68r5\""
Sep 10 17:33:37.967852 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.967818    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85030212-3063-4745-aa1b-4311abd98b13-openshift-service-ca\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:37.967852 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.967855    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-metrics-certs\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:37.968041 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.967884    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7p8\" (UniqueName: \"kubernetes.io/projected/85030212-3063-4745-aa1b-4311abd98b13-kube-api-access-bq7p8\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:37.968041 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:37.967921    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-maas-api-tls\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.068721 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.068687    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85030212-3063-4745-aa1b-4311abd98b13-openshift-service-ca\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.068721 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.068723    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-metrics-certs\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.068941 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.068751    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7p8\" (UniqueName: \"kubernetes.io/projected/85030212-3063-4745-aa1b-4311abd98b13-kube-api-access-bq7p8\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.068941 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.068766    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-maas-api-tls\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.069415 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.069389    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85030212-3063-4745-aa1b-4311abd98b13-openshift-service-ca\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.071291 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.071268    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-maas-api-tls\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.071390 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.071298    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-metrics-certs\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.077869 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.077847    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7p8\" (UniqueName: \"kubernetes.io/projected/85030212-3063-4745-aa1b-4311abd98b13-kube-api-access-bq7p8\") pod \"maas-api-84847f94bc-4gg6h\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") " pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.230070 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.229985    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:38.452534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.452499    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"]
Sep 10 17:33:38.455069 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.455051    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:38.458997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.458978    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 17:33:38.479497 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.479466    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"]
Sep 10 17:33:38.539568 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.539535    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"]
Sep 10 17:33:38.541748 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:33:38.541721    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85030212_3063_4745_aa1b_4311abd98b13.slice/crio-3c794fb07dc6bc8d395dd8fab5c4d759a6e4b78b8ec210f11f426669bb2153dc WatchSource:0}: Error finding container 3c794fb07dc6bc8d395dd8fab5c4d759a6e4b78b8ec210f11f426669bb2153dc: Status 404 returned error can't find the container with id 3c794fb07dc6bc8d395dd8fab5c4d759a6e4b78b8ec210f11f426669bb2153dc
Sep 10 17:33:38.573397 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.573359    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/008bb171-a493-4df9-b15e-e2d8a23ad60e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-s27sr\" (UID: \"008bb171-a493-4df9-b15e-e2d8a23ad60e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:38.573549 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.573404    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x482v\" (UniqueName: \"kubernetes.io/projected/008bb171-a493-4df9-b15e-e2d8a23ad60e-kube-api-access-x482v\") pod \"payload-pre-processing-59fbc47bf8-s27sr\" (UID: \"008bb171-a493-4df9-b15e-e2d8a23ad60e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:38.674935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.674888    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/008bb171-a493-4df9-b15e-e2d8a23ad60e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-s27sr\" (UID: \"008bb171-a493-4df9-b15e-e2d8a23ad60e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:38.674935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.674931    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x482v\" (UniqueName: \"kubernetes.io/projected/008bb171-a493-4df9-b15e-e2d8a23ad60e-kube-api-access-x482v\") pod \"payload-pre-processing-59fbc47bf8-s27sr\" (UID: \"008bb171-a493-4df9-b15e-e2d8a23ad60e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:38.675670 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.675643    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/008bb171-a493-4df9-b15e-e2d8a23ad60e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-s27sr\" (UID: \"008bb171-a493-4df9-b15e-e2d8a23ad60e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:38.688921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.688884    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x482v\" (UniqueName: \"kubernetes.io/projected/008bb171-a493-4df9-b15e-e2d8a23ad60e-kube-api-access-x482v\") pod \"payload-pre-processing-59fbc47bf8-s27sr\" (UID: \"008bb171-a493-4df9-b15e-e2d8a23ad60e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:38.765573 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:38.765532    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:39.054615 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:39.054462    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"]
Sep 10 17:33:39.056887 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:33:39.056862    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008bb171_a493_4df9_b15e_e2d8a23ad60e.slice/crio-4e5b7687b0d1022503f23d61c9e0735f2eb4cd13530076a9eeb89ad535a28624 WatchSource:0}: Error finding container 4e5b7687b0d1022503f23d61c9e0735f2eb4cd13530076a9eeb89ad535a28624: Status 404 returned error can't find the container with id 4e5b7687b0d1022503f23d61c9e0735f2eb4cd13530076a9eeb89ad535a28624
Sep 10 17:33:39.293301 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:39.293260    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr" event={"ID":"008bb171-a493-4df9-b15e-e2d8a23ad60e","Type":"ContainerStarted","Data":"4e5b7687b0d1022503f23d61c9e0735f2eb4cd13530076a9eeb89ad535a28624"}
Sep 10 17:33:39.294677 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:39.294649    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h" event={"ID":"85030212-3063-4745-aa1b-4311abd98b13","Type":"ContainerStarted","Data":"3c794fb07dc6bc8d395dd8fab5c4d759a6e4b78b8ec210f11f426669bb2153dc"}
Sep 10 17:33:41.307738 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:41.307701    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr" event={"ID":"008bb171-a493-4df9-b15e-e2d8a23ad60e","Type":"ContainerStarted","Data":"f5574f5837fa94f3808a1bd7e5268e8dea071ab5ca60549f53c6b9dcee2883d5"}
Sep 10 17:33:41.308154 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:41.307860    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:33:41.309305 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:41.309282    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h" event={"ID":"85030212-3063-4745-aa1b-4311abd98b13","Type":"ContainerStarted","Data":"ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4"}
Sep 10 17:33:41.309416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:41.309326    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:41.331775 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:41.331733    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr" podStartSLOduration=1.5658448520000001 podStartE2EDuration="3.331719613s" totalImagesPullingTime="1.765874761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:41.330126143 +0000 UTC m=+590.039720433" watchObservedRunningTime="2026-09-10 17:33:41.331719613 +0000 UTC m=+590.041313878"
Sep 10 17:33:41.361809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:41.356015    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h" podStartSLOduration=2.503445161 podStartE2EDuration="4.355995296s" totalImagesPullingTime="1.852550135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:41.35498298 +0000 UTC m=+590.064577245" watchObservedRunningTime="2026-09-10 17:33:41.355995296 +0000 UTC m=+590.065589564"
Sep 10 17:33:47.321210 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:47.321181    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:33:51.721287 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:51.721259    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:33:51.721730 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:51.721459    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:33:52.317290 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:33:52.317260    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s27sr"
Sep 10 17:34:22.004398 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.004310    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"]
Sep 10 17:34:22.007899 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.007878    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.012104 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.012076    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hjjt8\""
Sep 10 17:34:22.012218 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.012076    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 17:34:22.012218 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.012140    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 17:34:22.012218 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.012084    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 17:34:22.017320 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.017294    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"]
Sep 10 17:34:22.048018 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.047994    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.048134 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.048044    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b289abc-e9fd-40f7-aa71-d008dbb47072-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.048134 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.048085    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9w7b\" (UniqueName: \"kubernetes.io/projected/0b289abc-e9fd-40f7-aa71-d008dbb47072-kube-api-access-z9w7b\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.048134 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.048111    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.048334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.048137    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.048334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.048202    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.148858 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.148826    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.148858 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.148862    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b289abc-e9fd-40f7-aa71-d008dbb47072-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.149049 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.148887    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9w7b\" (UniqueName: \"kubernetes.io/projected/0b289abc-e9fd-40f7-aa71-d008dbb47072-kube-api-access-z9w7b\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.149049 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.149002    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.149049 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.149044    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.149165 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.149106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.149280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.149230    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.149479 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.149459    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.149479 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.149473    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.151164 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.151145    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b289abc-e9fd-40f7-aa71-d008dbb47072-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.151545 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.151526    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b289abc-e9fd-40f7-aa71-d008dbb47072-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.158035 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.158008    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9w7b\" (UniqueName: \"kubernetes.io/projected/0b289abc-e9fd-40f7-aa71-d008dbb47072-kube-api-access-z9w7b\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm\" (UID: \"0b289abc-e9fd-40f7-aa71-d008dbb47072\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.319646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.319615    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:22.592551 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.592526    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"]
Sep 10 17:34:22.594009 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:34:22.593981    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b289abc_e9fd_40f7_aa71_d008dbb47072.slice/crio-a75aa2d64dce3981a4ada90a0b6d16a2a01c1c850bcae2bd021546bdb15fe508 WatchSource:0}: Error finding container a75aa2d64dce3981a4ada90a0b6d16a2a01c1c850bcae2bd021546bdb15fe508: Status 404 returned error can't find the container with id a75aa2d64dce3981a4ada90a0b6d16a2a01c1c850bcae2bd021546bdb15fe508
Sep 10 17:34:22.595297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:22.595278    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:34:23.508359 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:23.508309    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm" event={"ID":"0b289abc-e9fd-40f7-aa71-d008dbb47072","Type":"ContainerStarted","Data":"a75aa2d64dce3981a4ada90a0b6d16a2a01c1c850bcae2bd021546bdb15fe508"}
Sep 10 17:34:24.514509 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:24.514472    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm" event={"ID":"0b289abc-e9fd-40f7-aa71-d008dbb47072","Type":"ContainerStarted","Data":"0396d4e082134bcace0a10b957890de611c2acd8c1a6059170e48e7e8ea6bf21"}
Sep 10 17:34:24.533154 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:24.533084    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm" podStartSLOduration=2.45888423 podStartE2EDuration="3.53307001s" totalImagesPullingTime="1.07418578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:24.532702543 +0000 UTC m=+633.242296811" watchObservedRunningTime="2026-09-10 17:34:24.53307001 +0000 UTC m=+633.242664275"
Sep 10 17:34:31.291908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.291877    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"]
Sep 10 17:34:31.295798 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.295777    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.298512 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.298491    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 17:34:31.310837 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.310802    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"]
Sep 10 17:34:31.327409 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.327382    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqzld\" (UniqueName: \"kubernetes.io/projected/05f50b85-67cc-462c-a477-dfa9e58948b7-kube-api-access-jqzld\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.327517 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.327422    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f50b85-67cc-462c-a477-dfa9e58948b7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.327517 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.327442    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.327591 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.327512    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.327591 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.327561    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.327591 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.327584    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.428430 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428398    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.428556 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428466    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzld\" (UniqueName: \"kubernetes.io/projected/05f50b85-67cc-462c-a477-dfa9e58948b7-kube-api-access-jqzld\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.428556 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428487    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f50b85-67cc-462c-a477-dfa9e58948b7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.428556 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428506    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.428556 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428524    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.428762 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428573    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.428896 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428872    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.428968 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428916    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.429030 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.428974    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.430834 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.430801    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f50b85-67cc-462c-a477-dfa9e58948b7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.431002 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.430985    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f50b85-67cc-462c-a477-dfa9e58948b7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.437751 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.437730    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzld\" (UniqueName: \"kubernetes.io/projected/05f50b85-67cc-462c-a477-dfa9e58948b7-kube-api-access-jqzld\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5xfzp\" (UID: \"05f50b85-67cc-462c-a477-dfa9e58948b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.610779 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.610698    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:31.881743 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:31.881717    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"]
Sep 10 17:34:31.884431 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:34:31.884406    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f50b85_67cc_462c_a477_dfa9e58948b7.slice/crio-5975771bba5bbc2fb863a3e520fd9a5a2da335a303e9f94c5ccbd1fbf590a10c WatchSource:0}: Error finding container 5975771bba5bbc2fb863a3e520fd9a5a2da335a303e9f94c5ccbd1fbf590a10c: Status 404 returned error can't find the container with id 5975771bba5bbc2fb863a3e520fd9a5a2da335a303e9f94c5ccbd1fbf590a10c
Sep 10 17:34:32.319786 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:32.319749    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:32.320201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:32.319828    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:32.332314 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:32.332281    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:32.554268 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:32.554220    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp" event={"ID":"05f50b85-67cc-462c-a477-dfa9e58948b7","Type":"ContainerStarted","Data":"02fea842525fc60a681f8eb57f4d22f31c03ffd7c021f7962a335d0a109a2f7e"}
Sep 10 17:34:32.554470 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:32.554295    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp" event={"ID":"05f50b85-67cc-462c-a477-dfa9e58948b7","Type":"ContainerStarted","Data":"5975771bba5bbc2fb863a3e520fd9a5a2da335a303e9f94c5ccbd1fbf590a10c"}
Sep 10 17:34:32.566224 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:32.566199    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm"
Sep 10 17:34:32.573216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:32.573159    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp" podStartSLOduration=1.3791821739999999 podStartE2EDuration="1.57313893s" totalImagesPullingTime="193.956756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:32.571160932 +0000 UTC m=+641.280755197" watchObservedRunningTime="2026-09-10 17:34:32.57313893 +0000 UTC m=+641.282733197"
Sep 10 17:34:38.115827 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.115793    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"]
Sep 10 17:34:38.119338 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.119314    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.122101 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.122074    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 17:34:38.129595 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.129569    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"]
Sep 10 17:34:38.189407 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.189374    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8335d9-ac37-4fe9-9909-782e118c2a22-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.189573 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.189415    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.189573 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.189440    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.189573 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.189534    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.189679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.189597    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwqj\" (UniqueName: \"kubernetes.io/projected/ea8335d9-ac37-4fe9-9909-782e118c2a22-kube-api-access-sfwqj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.189679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.189668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.290581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.290543    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8335d9-ac37-4fe9-9909-782e118c2a22-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.290581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.290582    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.290829 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.290608    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.290829 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.290657    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.290829 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.290683    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwqj\" (UniqueName: \"kubernetes.io/projected/ea8335d9-ac37-4fe9-9909-782e118c2a22-kube-api-access-sfwqj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.290829 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.290749    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.291063 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.291039    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.291117 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.291081    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.291155 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.291124    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.292993 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.292962    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8335d9-ac37-4fe9-9909-782e118c2a22-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.293244 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.293215    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8335d9-ac37-4fe9-9909-782e118c2a22-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.299690 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.299669    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwqj\" (UniqueName: \"kubernetes.io/projected/ea8335d9-ac37-4fe9-9909-782e118c2a22-kube-api-access-sfwqj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq\" (UID: \"ea8335d9-ac37-4fe9-9909-782e118c2a22\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.430171 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.430060    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:38.706406 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:38.706332    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"]
Sep 10 17:34:38.709803 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:34:38.709775    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8335d9_ac37_4fe9_9909_782e118c2a22.slice/crio-3a26e6791d39a48a3d3821bf3f17d86d79bca044d121b5aac6df2bb320b5fb63 WatchSource:0}: Error finding container 3a26e6791d39a48a3d3821bf3f17d86d79bca044d121b5aac6df2bb320b5fb63: Status 404 returned error can't find the container with id 3a26e6791d39a48a3d3821bf3f17d86d79bca044d121b5aac6df2bb320b5fb63
Sep 10 17:34:39.591054 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:39.591018    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq" event={"ID":"ea8335d9-ac37-4fe9-9909-782e118c2a22","Type":"ContainerStarted","Data":"f96870a56101983f62e9f60860771af3393f6a61fc1198fee93055e192e5a46a"}
Sep 10 17:34:39.591516 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:39.591073    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq" event={"ID":"ea8335d9-ac37-4fe9-9909-782e118c2a22","Type":"ContainerStarted","Data":"3a26e6791d39a48a3d3821bf3f17d86d79bca044d121b5aac6df2bb320b5fb63"}
Sep 10 17:34:39.613984 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:39.613927    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq" podStartSLOduration=1.424082271 podStartE2EDuration="1.61390919s" totalImagesPullingTime="189.826919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:39.612868533 +0000 UTC m=+648.322462810" watchObservedRunningTime="2026-09-10 17:34:39.61390919 +0000 UTC m=+648.323503478"
Sep 10 17:34:41.611041 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:41.611005    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:41.611490 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:41.611273    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:41.623626 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:41.623603    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:42.616864 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:42.616834    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5xfzp"
Sep 10 17:34:48.430629 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:48.430588    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:48.431080 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:48.430669    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:48.444838 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:48.444809    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:34:48.644027 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:34:48.643991    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq"
Sep 10 17:35:54.313913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.313830    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-645c68ff47-gnkzj"]
Sep 10 17:35:54.314437 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.314032    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-645c68ff47-gnkzj" podUID="b6676e15-3321-4ff4-bf33-349abf9b7727" containerName="authorino" containerID="cri-o://8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2" gracePeriod=30
Sep 10 17:35:54.568905 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.568850    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:35:54.654846 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.654803    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq9zn\" (UniqueName: \"kubernetes.io/projected/b6676e15-3321-4ff4-bf33-349abf9b7727-kube-api-access-sq9zn\") pod \"b6676e15-3321-4ff4-bf33-349abf9b7727\" (UID: \"b6676e15-3321-4ff4-bf33-349abf9b7727\") "
Sep 10 17:35:54.655016 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.654918    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6676e15-3321-4ff4-bf33-349abf9b7727-tls-cert\") pod \"b6676e15-3321-4ff4-bf33-349abf9b7727\" (UID: \"b6676e15-3321-4ff4-bf33-349abf9b7727\") "
Sep 10 17:35:54.656940 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.656904    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6676e15-3321-4ff4-bf33-349abf9b7727-kube-api-access-sq9zn" (OuterVolumeSpecName: "kube-api-access-sq9zn") pod "b6676e15-3321-4ff4-bf33-349abf9b7727" (UID: "b6676e15-3321-4ff4-bf33-349abf9b7727"). InnerVolumeSpecName "kube-api-access-sq9zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:35:54.664537 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.664511    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6676e15-3321-4ff4-bf33-349abf9b7727-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b6676e15-3321-4ff4-bf33-349abf9b7727" (UID: "b6676e15-3321-4ff4-bf33-349abf9b7727"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:35:54.755528 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.755494    2568 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6676e15-3321-4ff4-bf33-349abf9b7727-tls-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:35:54.755528 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.755522    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq9zn\" (UniqueName: \"kubernetes.io/projected/b6676e15-3321-4ff4-bf33-349abf9b7727-kube-api-access-sq9zn\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:35:54.960229 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.960118    2568 generic.go:356] "Generic (PLEG): container finished" podID="b6676e15-3321-4ff4-bf33-349abf9b7727" containerID="8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2" exitCode=0
Sep 10 17:35:54.960229 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.960180    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645c68ff47-gnkzj"
Sep 10 17:35:54.960229 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.960195    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-645c68ff47-gnkzj" event={"ID":"b6676e15-3321-4ff4-bf33-349abf9b7727","Type":"ContainerDied","Data":"8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2"}
Sep 10 17:35:54.960534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.960273    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-645c68ff47-gnkzj" event={"ID":"b6676e15-3321-4ff4-bf33-349abf9b7727","Type":"ContainerDied","Data":"7ab892316af4d9554235d6469cfb5dc661c06bb6abac1d09a6e8c7706ff53b08"}
Sep 10 17:35:54.960534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.960292    2568 scope.go:117] "RemoveContainer" containerID="8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2"
Sep 10 17:35:54.970218 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.970197    2568 scope.go:117] "RemoveContainer" containerID="8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2"
Sep 10 17:35:54.970514 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:35:54.970497    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2\": container with ID starting with 8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2 not found: ID does not exist" containerID="8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2"
Sep 10 17:35:54.970581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.970520    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2"} err="failed to get container status \"8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2\": rpc error: code = NotFound desc = could not find container \"8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2\": container with ID starting with 8394d4ea272454ce0da8300602c872e50b2f9af1d0c51b7a884de30a6d34a0c2 not found: ID does not exist"
Sep 10 17:35:54.990651 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.990622    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-645c68ff47-gnkzj"]
Sep 10 17:35:54.994395 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:54.994371    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-645c68ff47-gnkzj"]
Sep 10 17:35:55.764762 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:35:55.764729    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6676e15-3321-4ff4-bf33-349abf9b7727" path="/var/lib/kubelet/pods/b6676e15-3321-4ff4-bf33-349abf9b7727/volumes"
Sep 10 17:36:13.850557 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:13.850521    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7773-b7b35c"]
Sep 10 17:36:13.850957 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:13.850895    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6676e15-3321-4ff4-bf33-349abf9b7727" containerName="authorino"
Sep 10 17:36:13.850957 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:13.850907    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6676e15-3321-4ff4-bf33-349abf9b7727" containerName="authorino"
Sep 10 17:36:13.851028 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:13.850987    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6676e15-3321-4ff4-bf33-349abf9b7727" containerName="authorino"
Sep 10 17:36:13.855371 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:13.855355    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-b7b35c"
Sep 10 17:36:13.858385 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:13.858359    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p5m84\""
Sep 10 17:36:13.862646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:13.862618    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-b7b35c"]
Sep 10 17:36:13.914789 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:13.914756    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4bdj\" (UniqueName: \"kubernetes.io/projected/987cb25e-a2e1-4b57-a54c-d05328d1cbeb-kube-api-access-j4bdj\") pod \"test-curl-7773-b7b35c\" (UID: \"987cb25e-a2e1-4b57-a54c-d05328d1cbeb\") " pod="opendatahub/test-curl-7773-b7b35c"
Sep 10 17:36:14.015326 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:14.015284    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4bdj\" (UniqueName: \"kubernetes.io/projected/987cb25e-a2e1-4b57-a54c-d05328d1cbeb-kube-api-access-j4bdj\") pod \"test-curl-7773-b7b35c\" (UID: \"987cb25e-a2e1-4b57-a54c-d05328d1cbeb\") " pod="opendatahub/test-curl-7773-b7b35c"
Sep 10 17:36:14.023965 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:14.023931    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4bdj\" (UniqueName: \"kubernetes.io/projected/987cb25e-a2e1-4b57-a54c-d05328d1cbeb-kube-api-access-j4bdj\") pod \"test-curl-7773-b7b35c\" (UID: \"987cb25e-a2e1-4b57-a54c-d05328d1cbeb\") " pod="opendatahub/test-curl-7773-b7b35c"
Sep 10 17:36:14.166742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:14.166666    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-b7b35c"
Sep 10 17:36:14.440160 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:14.440133    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-b7b35c"]
Sep 10 17:36:14.442346 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:14.442305    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod987cb25e_a2e1_4b57_a54c_d05328d1cbeb.slice/crio-c4efa52b951cdd9039b26f05131550200cd7c33703c8640b79b68b7b131a75ee WatchSource:0}: Error finding container c4efa52b951cdd9039b26f05131550200cd7c33703c8640b79b68b7b131a75ee: Status 404 returned error can't find the container with id c4efa52b951cdd9039b26f05131550200cd7c33703c8640b79b68b7b131a75ee
Sep 10 17:36:15.057664 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:15.057627    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-b7b35c" event={"ID":"987cb25e-a2e1-4b57-a54c-d05328d1cbeb","Type":"ContainerStarted","Data":"c4efa52b951cdd9039b26f05131550200cd7c33703c8640b79b68b7b131a75ee"}
Sep 10 17:36:16.032577 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.032543    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"]
Sep 10 17:36:16.037229 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.037196    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.040079 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.040050    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0d25fa-serving-cert\""
Sep 10 17:36:16.047771 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.047742    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"]
Sep 10 17:36:16.128783 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.128750    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8kxw\" (UniqueName: \"kubernetes.io/projected/a9815a30-c0f0-4280-a046-318d9b7fd340-kube-api-access-m8kxw\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.128783 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.128786    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9815a30-c0f0-4280-a046-318d9b7fd340-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.129209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.128880    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.129209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.128921    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-metrics-certs\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.229665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.229623    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-metrics-certs\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.229862 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.229714    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxw\" (UniqueName: \"kubernetes.io/projected/a9815a30-c0f0-4280-a046-318d9b7fd340-kube-api-access-m8kxw\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.229862 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.229738    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9815a30-c0f0-4280-a046-318d9b7fd340-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.229862 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.229793    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.230030 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:16.229910    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-serving-cert: secret "maas-api-e2e-inf-a-0d25fa-serving-cert" not found
Sep 10 17:36:16.230030 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:16.229973    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls podName:a9815a30-c0f0-4280-a046-318d9b7fd340 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:16.729952452 +0000 UTC m=+745.439546716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls") pod "maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5" (UID: "a9815a30-c0f0-4280-a046-318d9b7fd340") : secret "maas-api-e2e-inf-a-0d25fa-serving-cert" not found
Sep 10 17:36:16.230430 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.230408    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9815a30-c0f0-4280-a046-318d9b7fd340-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.232276 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.232253    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-metrics-certs\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.251045 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.251016    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8kxw\" (UniqueName: \"kubernetes.io/projected/a9815a30-c0f0-4280-a046-318d9b7fd340-kube-api-access-m8kxw\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.732939 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.732901    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.736545 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.736515    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:16.952398 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:16.952364    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:17.239605 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:17.239579    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"]
Sep 10 17:36:17.241640 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:17.241595    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9815a30_c0f0_4280_a046_318d9b7fd340.slice/crio-fe33a97fc93d3e0d76ff029a5de8202b9a0f53753042eb90a479103a615006c5 WatchSource:0}: Error finding container fe33a97fc93d3e0d76ff029a5de8202b9a0f53753042eb90a479103a615006c5: Status 404 returned error can't find the container with id fe33a97fc93d3e0d76ff029a5de8202b9a0f53753042eb90a479103a615006c5
Sep 10 17:36:18.080382 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:18.080345    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5" event={"ID":"a9815a30-c0f0-4280-a046-318d9b7fd340","Type":"ContainerStarted","Data":"fe33a97fc93d3e0d76ff029a5de8202b9a0f53753042eb90a479103a615006c5"}
Sep 10 17:36:19.086740 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.086705    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-b7b35c" event={"ID":"987cb25e-a2e1-4b57-a54c-d05328d1cbeb","Type":"ContainerStarted","Data":"5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5"}
Sep 10 17:36:19.088348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.088321    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5" event={"ID":"a9815a30-c0f0-4280-a046-318d9b7fd340","Type":"ContainerStarted","Data":"6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e"}
Sep 10 17:36:19.088479 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.088403    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:19.103295 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.103245    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7773-b7b35c" podStartSLOduration=2.497060325 podStartE2EDuration="6.103217424s" totalImagesPullingTime="3.606157099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:19.101622218 +0000 UTC m=+747.811216484" watchObservedRunningTime="2026-09-10 17:36:19.103217424 +0000 UTC m=+747.812811711"
Sep 10 17:36:19.121730 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.121674    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5" podStartSLOduration=2.314679668 podStartE2EDuration="3.121653583s" totalImagesPullingTime="806.973915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:19.12131698 +0000 UTC m=+747.830911247" watchObservedRunningTime="2026-09-10 17:36:19.121653583 +0000 UTC m=+747.831247852"
Sep 10 17:36:19.694865 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.694828    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7773-b7b35c"]
Sep 10 17:36:19.698972 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.698937    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7773-b7b35c"]
Sep 10 17:36:19.821635 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.821590    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7773-c4052b"]
Sep 10 17:36:19.821996 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.821984    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987cb25e-a2e1-4b57-a54c-d05328d1cbeb" containerName="test-curl-7773-b7b35c"
Sep 10 17:36:19.822039 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.821997    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="987cb25e-a2e1-4b57-a54c-d05328d1cbeb" containerName="test-curl-7773-b7b35c"
Sep 10 17:36:19.822071 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.822062    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="987cb25e-a2e1-4b57-a54c-d05328d1cbeb" containerName="test-curl-7773-b7b35c"
Sep 10 17:36:19.825222 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.825207    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-c4052b"
Sep 10 17:36:19.830584 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.830556    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-c4052b"]
Sep 10 17:36:19.864022 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.863980    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59f2f\" (UniqueName: \"kubernetes.io/projected/837d57d8-9fc0-4918-838e-5ab381c22028-kube-api-access-59f2f\") pod \"test-curl-7773-c4052b\" (UID: \"837d57d8-9fc0-4918-838e-5ab381c22028\") " pod="opendatahub/test-curl-7773-c4052b"
Sep 10 17:36:19.964566 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.964524    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59f2f\" (UniqueName: \"kubernetes.io/projected/837d57d8-9fc0-4918-838e-5ab381c22028-kube-api-access-59f2f\") pod \"test-curl-7773-c4052b\" (UID: \"837d57d8-9fc0-4918-838e-5ab381c22028\") " pod="opendatahub/test-curl-7773-c4052b"
Sep 10 17:36:19.973441 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:19.973414    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59f2f\" (UniqueName: \"kubernetes.io/projected/837d57d8-9fc0-4918-838e-5ab381c22028-kube-api-access-59f2f\") pod \"test-curl-7773-c4052b\" (UID: \"837d57d8-9fc0-4918-838e-5ab381c22028\") " pod="opendatahub/test-curl-7773-c4052b"
Sep 10 17:36:20.093578 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:20.093491    2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7773-b7b35c" container="test-curl-7773-b7b35c"
Sep 10 17:36:20.135740 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:20.135701    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-c4052b"
Sep 10 17:36:20.417400 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:20.417377    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-c4052b"]
Sep 10 17:36:20.419386 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:20.419358    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod837d57d8_9fc0_4918_838e_5ab381c22028.slice/crio-7ad0838ae390cbc18af2720486d11b89d5f1b7a6162dea6e55b75ee688d79f33 WatchSource:0}: Error finding container 7ad0838ae390cbc18af2720486d11b89d5f1b7a6162dea6e55b75ee688d79f33: Status 404 returned error can't find the container with id 7ad0838ae390cbc18af2720486d11b89d5f1b7a6162dea6e55b75ee688d79f33
Sep 10 17:36:21.100090 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:21.100052    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-c4052b" event={"ID":"837d57d8-9fc0-4918-838e-5ab381c22028","Type":"ContainerStarted","Data":"124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a"}
Sep 10 17:36:21.100563 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:21.100101    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-c4052b" event={"ID":"837d57d8-9fc0-4918-838e-5ab381c22028","Type":"ContainerStarted","Data":"7ad0838ae390cbc18af2720486d11b89d5f1b7a6162dea6e55b75ee688d79f33"}
Sep 10 17:36:21.100563 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:21.100126    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7773-b7b35c" podUID="987cb25e-a2e1-4b57-a54c-d05328d1cbeb" containerName="test-curl-7773-b7b35c" containerID="cri-o://5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5" gracePeriod=2
Sep 10 17:36:21.118200 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:21.118150    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7773-c4052b" podStartSLOduration=2.11813629 podStartE2EDuration="2.11813629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:21.116840081 +0000 UTC m=+749.826434350" watchObservedRunningTime="2026-09-10 17:36:21.11813629 +0000 UTC m=+749.827730557"
Sep 10 17:36:21.706396 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:21.706359    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7773-c4052b"]
Sep 10 17:36:21.711738 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:21.711709    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7773-c4052b"]
Sep 10 17:36:22.105643 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:22.105605    2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7773-c4052b" container="test-curl-7773-c4052b"
Sep 10 17:36:23.110367 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:23.110332    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7773-c4052b" podUID="837d57d8-9fc0-4918-838e-5ab381c22028" containerName="test-curl-7773-c4052b" containerID="cri-o://124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a" gracePeriod=2
Sep 10 17:36:23.268118 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:23.268091    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-b7b35c"
Sep 10 17:36:23.295291 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:23.295262    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4bdj\" (UniqueName: \"kubernetes.io/projected/987cb25e-a2e1-4b57-a54c-d05328d1cbeb-kube-api-access-j4bdj\") pod \"987cb25e-a2e1-4b57-a54c-d05328d1cbeb\" (UID: \"987cb25e-a2e1-4b57-a54c-d05328d1cbeb\") "
Sep 10 17:36:23.297389 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:23.297357    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987cb25e-a2e1-4b57-a54c-d05328d1cbeb-kube-api-access-j4bdj" (OuterVolumeSpecName: "kube-api-access-j4bdj") pod "987cb25e-a2e1-4b57-a54c-d05328d1cbeb" (UID: "987cb25e-a2e1-4b57-a54c-d05328d1cbeb"). InnerVolumeSpecName "kube-api-access-j4bdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:36:23.396818 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:23.396731    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4bdj\" (UniqueName: \"kubernetes.io/projected/987cb25e-a2e1-4b57-a54c-d05328d1cbeb-kube-api-access-j4bdj\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:36:23.763391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:23.763319    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987cb25e-a2e1-4b57-a54c-d05328d1cbeb" path="/var/lib/kubelet/pods/987cb25e-a2e1-4b57-a54c-d05328d1cbeb/volumes"
Sep 10 17:36:24.116613 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.116578    2568 generic.go:356] "Generic (PLEG): container finished" podID="987cb25e-a2e1-4b57-a54c-d05328d1cbeb" containerID="5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5" exitCode=137
Sep 10 17:36:24.116613 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.116618    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-b7b35c"
Sep 10 17:36:24.117079 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.116653    2568 scope.go:117] "RemoveContainer" containerID="5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5"
Sep 10 17:36:24.124801 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.124779    2568 scope.go:117] "RemoveContainer" containerID="5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5"
Sep 10 17:36:24.125091 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:24.125068    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5\": container with ID starting with 5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5 not found: ID does not exist" containerID="5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5"
Sep 10 17:36:24.125164 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.125099    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5"} err="failed to get container status \"5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5\": rpc error: code = NotFound desc = could not find container \"5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5\": container with ID starting with 5e7dcc88e4e71cdce27818894fe34485a8ddea3bbb20488f2f4f7cb2e89135b5 not found: ID does not exist"
Sep 10 17:36:24.619344 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.619310    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"]
Sep 10 17:36:24.619908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.619888    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837d57d8-9fc0-4918-838e-5ab381c22028" containerName="test-curl-7773-c4052b"
Sep 10 17:36:24.619955 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.619913    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="837d57d8-9fc0-4918-838e-5ab381c22028" containerName="test-curl-7773-c4052b"
Sep 10 17:36:24.620015 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.620002    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="837d57d8-9fc0-4918-838e-5ab381c22028" containerName="test-curl-7773-c4052b"
Sep 10 17:36:24.624549 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.624526    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.627281 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.627226    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-64df4d9a-serving-cert\""
Sep 10 17:36:24.631490 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.631464    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"]
Sep 10 17:36:24.710032 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.709997    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660991f-2a55-4149-aaa0-f8269c95486e-openshift-service-ca\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.710032 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.710038    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-metrics-certs\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.710289 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.710079    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-maas-api-tls\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.710289 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.710165    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6rl\" (UniqueName: \"kubernetes.io/projected/2660991f-2a55-4149-aaa0-f8269c95486e-kube-api-access-gb6rl\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.811340 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.811308    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6rl\" (UniqueName: \"kubernetes.io/projected/2660991f-2a55-4149-aaa0-f8269c95486e-kube-api-access-gb6rl\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.811722 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.811361    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660991f-2a55-4149-aaa0-f8269c95486e-openshift-service-ca\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.811722 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.811519    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-metrics-certs\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.811722 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.811631    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-maas-api-tls\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.812125 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.812097    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660991f-2a55-4149-aaa0-f8269c95486e-openshift-service-ca\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.814001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.813980    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-maas-api-tls\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.814085 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.814053    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-metrics-certs\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.823156 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.823135    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6rl\" (UniqueName: \"kubernetes.io/projected/2660991f-2a55-4149-aaa0-f8269c95486e-kube-api-access-gb6rl\") pod \"maas-api-e2e-ait-64df4d9a-84d645d956-75d6n\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:24.938334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:24.938229    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:25.100001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.099972    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:36:25.221445 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.221418    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"]
Sep 10 17:36:25.223316 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:25.223286    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2660991f_2a55_4149_aaa0_f8269c95486e.slice/crio-b74af79b8f21e134c98454ce36c52f9f910584e2179f1e3bd7948868f9ab4742 WatchSource:0}: Error finding container b74af79b8f21e134c98454ce36c52f9f910584e2179f1e3bd7948868f9ab4742: Status 404 returned error can't find the container with id b74af79b8f21e134c98454ce36c52f9f910584e2179f1e3bd7948868f9ab4742
Sep 10 17:36:25.255149 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.255126    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-c4052b"
Sep 10 17:36:25.279853 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.279825    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"]
Sep 10 17:36:25.283520 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.283496    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:25.286503 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.286481    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-64df4d9a-dockercfg-q6b8c\""
Sep 10 17:36:25.286855 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.286835    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-64df4d9a\""
Sep 10 17:36:25.293352 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.293330    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"]
Sep 10 17:36:25.297161 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.297138    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:25.297444 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.297425    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"]
Sep 10 17:36:25.310568 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.310535    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"]
Sep 10 17:36:25.316216 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.316190    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59f2f\" (UniqueName: \"kubernetes.io/projected/837d57d8-9fc0-4918-838e-5ab381c22028-kube-api-access-59f2f\") pod \"837d57d8-9fc0-4918-838e-5ab381c22028\" (UID: \"837d57d8-9fc0-4918-838e-5ab381c22028\") "
Sep 10 17:36:25.316413 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.316391    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5chb\" (UniqueName: \"kubernetes.io/projected/7330b0f2-07ee-4972-99eb-e248d850e7a3-kube-api-access-v5chb\") pod \"payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d\" (UID: \"7330b0f2-07ee-4972-99eb-e248d850e7a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:25.316579 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.316562    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330b0f2-07ee-4972-99eb-e248d850e7a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d\" (UID: \"7330b0f2-07ee-4972-99eb-e248d850e7a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:25.318380 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.318357    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837d57d8-9fc0-4918-838e-5ab381c22028-kube-api-access-59f2f" (OuterVolumeSpecName: "kube-api-access-59f2f") pod "837d57d8-9fc0-4918-838e-5ab381c22028" (UID: "837d57d8-9fc0-4918-838e-5ab381c22028"). InnerVolumeSpecName "kube-api-access-59f2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:36:25.417086 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.417047    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2xk\" (UniqueName: \"kubernetes.io/projected/68cf7190-b29a-4779-ae19-9ad4d48b9dba-kube-api-access-cq2xk\") pod \"payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n\" (UID: \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\") " pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:25.417309 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.417128    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330b0f2-07ee-4972-99eb-e248d850e7a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d\" (UID: \"7330b0f2-07ee-4972-99eb-e248d850e7a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:25.417309 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.417170    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5chb\" (UniqueName: \"kubernetes.io/projected/7330b0f2-07ee-4972-99eb-e248d850e7a3-kube-api-access-v5chb\") pod \"payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d\" (UID: \"7330b0f2-07ee-4972-99eb-e248d850e7a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:25.417309 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.417202    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cf7190-b29a-4779-ae19-9ad4d48b9dba-plugins-config-volume\") pod \"payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n\" (UID: \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\") " pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:25.417309 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.417302    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59f2f\" (UniqueName: \"kubernetes.io/projected/837d57d8-9fc0-4918-838e-5ab381c22028-kube-api-access-59f2f\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:36:25.417853 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.417830    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330b0f2-07ee-4972-99eb-e248d850e7a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d\" (UID: \"7330b0f2-07ee-4972-99eb-e248d850e7a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:25.426533 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.426509    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5chb\" (UniqueName: \"kubernetes.io/projected/7330b0f2-07ee-4972-99eb-e248d850e7a3-kube-api-access-v5chb\") pod \"payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d\" (UID: \"7330b0f2-07ee-4972-99eb-e248d850e7a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:25.518378 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.518279    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2xk\" (UniqueName: \"kubernetes.io/projected/68cf7190-b29a-4779-ae19-9ad4d48b9dba-kube-api-access-cq2xk\") pod \"payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n\" (UID: \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\") " pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:25.518378 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.518362    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cf7190-b29a-4779-ae19-9ad4d48b9dba-plugins-config-volume\") pod \"payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n\" (UID: \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\") " pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:25.519149 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.519121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cf7190-b29a-4779-ae19-9ad4d48b9dba-plugins-config-volume\") pod \"payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n\" (UID: \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\") " pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:25.527081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.527054    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2xk\" (UniqueName: \"kubernetes.io/projected/68cf7190-b29a-4779-ae19-9ad4d48b9dba-kube-api-access-cq2xk\") pod \"payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n\" (UID: \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\") " pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:25.594370 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.594340    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:25.607124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.607099    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:25.764178 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.764144    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837d57d8-9fc0-4918-838e-5ab381c22028" path="/var/lib/kubelet/pods/837d57d8-9fc0-4918-838e-5ab381c22028/volumes"
Sep 10 17:36:25.904608 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.904561    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"]
Sep 10 17:36:25.905540 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:25.905511    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7330b0f2_07ee_4972_99eb_e248d850e7a3.slice/crio-2b117715c410cf8cab7e6e1fa20cb5f113cf8942538f238ed446092bf5352775 WatchSource:0}: Error finding container 2b117715c410cf8cab7e6e1fa20cb5f113cf8942538f238ed446092bf5352775: Status 404 returned error can't find the container with id 2b117715c410cf8cab7e6e1fa20cb5f113cf8942538f238ed446092bf5352775
Sep 10 17:36:25.927030 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:25.927002    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"]
Sep 10 17:36:25.928973 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:25.928946    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cf7190_b29a_4779_ae19_9ad4d48b9dba.slice/crio-1e7168ffc279f6271899671c64db22071231ceb5740cadd8750b9c1436d9cb94 WatchSource:0}: Error finding container 1e7168ffc279f6271899671c64db22071231ceb5740cadd8750b9c1436d9cb94: Status 404 returned error can't find the container with id 1e7168ffc279f6271899671c64db22071231ceb5740cadd8750b9c1436d9cb94
Sep 10 17:36:26.130507 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.130404    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d" event={"ID":"7330b0f2-07ee-4972-99eb-e248d850e7a3","Type":"ContainerStarted","Data":"af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407"}
Sep 10 17:36:26.130507 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.130455    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d" event={"ID":"7330b0f2-07ee-4972-99eb-e248d850e7a3","Type":"ContainerStarted","Data":"2b117715c410cf8cab7e6e1fa20cb5f113cf8942538f238ed446092bf5352775"}
Sep 10 17:36:26.130742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.130510    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:26.131753 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.131732    2568 generic.go:356] "Generic (PLEG): container finished" podID="837d57d8-9fc0-4918-838e-5ab381c22028" containerID="124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a" exitCode=137
Sep 10 17:36:26.131844 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.131794    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-c4052b"
Sep 10 17:36:26.131898 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.131852    2568 scope.go:117] "RemoveContainer" containerID="124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a"
Sep 10 17:36:26.133834 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.133807    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n" event={"ID":"68cf7190-b29a-4779-ae19-9ad4d48b9dba","Type":"ContainerStarted","Data":"54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852"}
Sep 10 17:36:26.133967 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.133947    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n" event={"ID":"68cf7190-b29a-4779-ae19-9ad4d48b9dba","Type":"ContainerStarted","Data":"1e7168ffc279f6271899671c64db22071231ceb5740cadd8750b9c1436d9cb94"}
Sep 10 17:36:26.135574 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.135553    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n" event={"ID":"2660991f-2a55-4149-aaa0-f8269c95486e","Type":"ContainerStarted","Data":"b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa"}
Sep 10 17:36:26.135674 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.135587    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n" event={"ID":"2660991f-2a55-4149-aaa0-f8269c95486e","Type":"ContainerStarted","Data":"b74af79b8f21e134c98454ce36c52f9f910584e2179f1e3bd7948868f9ab4742"}
Sep 10 17:36:26.135723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.135684    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:26.140838 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.140814    2568 scope.go:117] "RemoveContainer" containerID="124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a"
Sep 10 17:36:26.141138 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:26.141111    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a\": container with ID starting with 124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a not found: ID does not exist" containerID="124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a"
Sep 10 17:36:26.141253 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.141147    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a"} err="failed to get container status \"124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a\": rpc error: code = NotFound desc = could not find container \"124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a\": container with ID starting with 124f8d61412d750aeee3d7cffea737aa60e628c45ff2dd44d7d6ecb92485a50a not found: ID does not exist"
Sep 10 17:36:26.154404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.154365    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d" podStartSLOduration=1.154353944 podStartE2EDuration="1.154353944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:26.151590164 +0000 UTC m=+754.861184430" watchObservedRunningTime="2026-09-10 17:36:26.154353944 +0000 UTC m=+754.863948210"
Sep 10 17:36:26.169499 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.169447    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n" podStartSLOduration=1.790913634 podStartE2EDuration="2.169429764s" totalImagesPullingTime="378.51613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:26.168807249 +0000 UTC m=+754.878401554" watchObservedRunningTime="2026-09-10 17:36:26.169429764 +0000 UTC m=+754.879024031"
Sep 10 17:36:26.188130 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:26.188075    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n" podStartSLOduration=1.188058102 podStartE2EDuration="1.188058102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:26.186706385 +0000 UTC m=+754.896300659" watchObservedRunningTime="2026-09-10 17:36:26.188058102 +0000 UTC m=+754.897652368"
Sep 10 17:36:27.109215 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.109059    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"]
Sep 10 17:36:27.114170 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.114148    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.116868 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.116843    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-794622-serving-cert\""
Sep 10 17:36:27.125478 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.125420    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"]
Sep 10 17:36:27.147533 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.147509    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:27.236129 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.236095    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.236358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.236209    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-metrics-certs\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.236358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.236279    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqr7d\" (UniqueName: \"kubernetes.io/projected/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-kube-api-access-bqr7d\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.236358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.236341    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-openshift-service-ca\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.337736 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.337701    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.337886 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.337782    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-metrics-certs\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.337886 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.337818    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqr7d\" (UniqueName: \"kubernetes.io/projected/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-kube-api-access-bqr7d\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.337886 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:27.337836    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-794622-serving-cert: secret "maas-api-e2e-autores-794622-serving-cert" not found
Sep 10 17:36:27.337886 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.337854    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-openshift-service-ca\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.338051 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:27.337921    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls podName:d6892ca0-8bb8-48c7-9b9d-952632b5e42c nodeName:}" failed. No retries permitted until 2026-09-10 17:36:27.837904836 +0000 UTC m=+756.547499085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls") pod "maas-api-e2e-autores-794622-c8964fd77-cfrts" (UID: "d6892ca0-8bb8-48c7-9b9d-952632b5e42c") : secret "maas-api-e2e-autores-794622-serving-cert" not found
Sep 10 17:36:27.338713 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.338693    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-openshift-service-ca\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.340468 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.340441    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-metrics-certs\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.349363 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.349336    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqr7d\" (UniqueName: \"kubernetes.io/projected/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-kube-api-access-bqr7d\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.730624 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.730595    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"]
Sep 10 17:36:27.734200 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.734179    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:27.736738 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.736715    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-794622\""
Sep 10 17:36:27.737266 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.737247    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-794622-dockercfg-8k85q\""
Sep 10 17:36:27.751629 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.751604    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"]
Sep 10 17:36:27.841537 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.841500    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1963dba5-19a1-4c21-b1a2-5af18c06261a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8\" (UID: \"1963dba5-19a1-4c21-b1a2-5af18c06261a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:27.841727 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.841560    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.841727 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.841649    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqpdm\" (UniqueName: \"kubernetes.io/projected/1963dba5-19a1-4c21-b1a2-5af18c06261a-kube-api-access-wqpdm\") pod \"payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8\" (UID: \"1963dba5-19a1-4c21-b1a2-5af18c06261a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:27.844142 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.844118    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls\") pod \"maas-api-e2e-autores-794622-c8964fd77-cfrts\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:27.944150 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.942627    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqpdm\" (UniqueName: \"kubernetes.io/projected/1963dba5-19a1-4c21-b1a2-5af18c06261a-kube-api-access-wqpdm\") pod \"payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8\" (UID: \"1963dba5-19a1-4c21-b1a2-5af18c06261a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:27.944150 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.942722    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1963dba5-19a1-4c21-b1a2-5af18c06261a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8\" (UID: \"1963dba5-19a1-4c21-b1a2-5af18c06261a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:27.944150 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.943630    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1963dba5-19a1-4c21-b1a2-5af18c06261a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8\" (UID: \"1963dba5-19a1-4c21-b1a2-5af18c06261a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:27.961065 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:27.961032    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqpdm\" (UniqueName: \"kubernetes.io/projected/1963dba5-19a1-4c21-b1a2-5af18c06261a-kube-api-access-wqpdm\") pod \"payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8\" (UID: \"1963dba5-19a1-4c21-b1a2-5af18c06261a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:28.027403 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:28.027378    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:28.047454 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:28.047424    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:28.310755 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:28.310730    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"]
Sep 10 17:36:28.313497 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:28.313467    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6892ca0_8bb8_48c7_9b9d_952632b5e42c.slice/crio-b71ffb53a7635651b4736622f00bbd49e40c83d794ccc751f7ec44dc024ffbae WatchSource:0}: Error finding container b71ffb53a7635651b4736622f00bbd49e40c83d794ccc751f7ec44dc024ffbae: Status 404 returned error can't find the container with id b71ffb53a7635651b4736622f00bbd49e40c83d794ccc751f7ec44dc024ffbae
Sep 10 17:36:28.331774 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:28.331751    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"]
Sep 10 17:36:28.332925 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:28.332904    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1963dba5_19a1_4c21_b1a2_5af18c06261a.slice/crio-8e77f2625c4e946e20b889c76596204eab875bb10abd9b441173442d21a50e5e WatchSource:0}: Error finding container 8e77f2625c4e946e20b889c76596204eab875bb10abd9b441173442d21a50e5e: Status 404 returned error can't find the container with id 8e77f2625c4e946e20b889c76596204eab875bb10abd9b441173442d21a50e5e
Sep 10 17:36:29.159915 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:29.159878    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts" event={"ID":"d6892ca0-8bb8-48c7-9b9d-952632b5e42c","Type":"ContainerStarted","Data":"04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312"}
Sep 10 17:36:29.160123 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:29.159968    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts" event={"ID":"d6892ca0-8bb8-48c7-9b9d-952632b5e42c","Type":"ContainerStarted","Data":"b71ffb53a7635651b4736622f00bbd49e40c83d794ccc751f7ec44dc024ffbae"}
Sep 10 17:36:29.160123 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:29.159996    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:29.161672 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:29.161645    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8" event={"ID":"1963dba5-19a1-4c21-b1a2-5af18c06261a","Type":"ContainerStarted","Data":"c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7"}
Sep 10 17:36:29.161991 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:29.161687    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8" event={"ID":"1963dba5-19a1-4c21-b1a2-5af18c06261a","Type":"ContainerStarted","Data":"8e77f2625c4e946e20b889c76596204eab875bb10abd9b441173442d21a50e5e"}
Sep 10 17:36:29.161991 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:29.161728    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:29.178537 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:29.178494    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts" podStartSLOduration=1.834995602 podStartE2EDuration="2.178478909s" totalImagesPullingTime="343.483307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:29.176226961 +0000 UTC m=+757.885821227" watchObservedRunningTime="2026-09-10 17:36:29.178478909 +0000 UTC m=+757.888073210"
Sep 10 17:36:29.194368 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:29.194293    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8" podStartSLOduration=2.194279572 podStartE2EDuration="2.194279572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:29.193194328 +0000 UTC m=+757.902788596" watchObservedRunningTime="2026-09-10 17:36:29.194279572 +0000 UTC m=+757.903873838"
Sep 10 17:36:32.153812 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:32.153784    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:36:35.172018 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:35.171988    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:36:37.149934 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:37.149895    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:36:38.154951 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:38.154916    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:36:40.169007 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:40.168977    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:36:47.711276 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.711224    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"]
Sep 10 17:36:47.714941 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.714919    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.717786 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.717769    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2f1093-serving-cert\""
Sep 10 17:36:47.725304 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.725281    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"]
Sep 10 17:36:47.738542 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.738520    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-metrics-certs\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.738542 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.738548    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.738678 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.738582    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlssn\" (UniqueName: \"kubernetes.io/projected/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-kube-api-access-vlssn\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.738678 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.738615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.839917 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.839883    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-metrics-certs\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.840081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.839927    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.840081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.839968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlssn\" (UniqueName: \"kubernetes.io/projected/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-kube-api-access-vlssn\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.840081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.840012    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.840190 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:47.840073    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-serving-cert: secret "maas-api-e2e-inf-b-2f1093-serving-cert" not found
Sep 10 17:36:47.840190 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:36:47.840141    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls podName:4f2b010b-fbd5-4059-a14d-0a5a52dd4354 nodeName:}" failed. No retries permitted until 2026-09-10 17:36:48.340121527 +0000 UTC m=+777.049715777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls") pod "maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z" (UID: "4f2b010b-fbd5-4059-a14d-0a5a52dd4354") : secret "maas-api-e2e-inf-b-2f1093-serving-cert" not found
Sep 10 17:36:47.840857 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.840831    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.842726 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.842692    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-metrics-certs\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:47.849707 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:47.849683    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlssn\" (UniqueName: \"kubernetes.io/projected/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-kube-api-access-vlssn\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:48.344120 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.344080    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:48.346742 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.346705    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls\") pod \"maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:48.545965 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.545918    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"]
Sep 10 17:36:48.552039 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.552007    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:48.555844 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.555811    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2f1093-dockercfg-nv9kc\""
Sep 10 17:36:48.556004 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.555905    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2f1093\""
Sep 10 17:36:48.563197 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.563171    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"]
Sep 10 17:36:48.626624 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.626552    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:48.647085 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.647055    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439ea1b8-0bc2-438b-a259-8ce664f5fc05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls\" (UID: \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:48.647316 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.647289    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4vcx\" (UniqueName: \"kubernetes.io/projected/439ea1b8-0bc2-438b-a259-8ce664f5fc05-kube-api-access-j4vcx\") pod \"payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls\" (UID: \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:48.747746 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.747713    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439ea1b8-0bc2-438b-a259-8ce664f5fc05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls\" (UID: \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:48.748207 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.747772    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4vcx\" (UniqueName: \"kubernetes.io/projected/439ea1b8-0bc2-438b-a259-8ce664f5fc05-kube-api-access-j4vcx\") pod \"payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls\" (UID: \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:48.748602 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.748575    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439ea1b8-0bc2-438b-a259-8ce664f5fc05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls\" (UID: \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:48.757599 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.757574    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4vcx\" (UniqueName: \"kubernetes.io/projected/439ea1b8-0bc2-438b-a259-8ce664f5fc05-kube-api-access-j4vcx\") pod \"payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls\" (UID: \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:48.865867 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.865834    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:48.910212 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:48.910178    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"]
Sep 10 17:36:48.912212 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:48.912157    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2b010b_fbd5_4059_a14d_0a5a52dd4354.slice/crio-044899384f00541699acabf1f0e82c39b6f2a7bae26bb19aad3f66fad13f969e WatchSource:0}: Error finding container 044899384f00541699acabf1f0e82c39b6f2a7bae26bb19aad3f66fad13f969e: Status 404 returned error can't find the container with id 044899384f00541699acabf1f0e82c39b6f2a7bae26bb19aad3f66fad13f969e
Sep 10 17:36:49.145014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:49.144980    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"]
Sep 10 17:36:49.147275 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:36:49.147247    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439ea1b8_0bc2_438b_a259_8ce664f5fc05.slice/crio-c3a10b5a1bbb83225a6a7eb374485646d97b138ad74a0bb5c6357af18b2a3b1b WatchSource:0}: Error finding container c3a10b5a1bbb83225a6a7eb374485646d97b138ad74a0bb5c6357af18b2a3b1b: Status 404 returned error can't find the container with id c3a10b5a1bbb83225a6a7eb374485646d97b138ad74a0bb5c6357af18b2a3b1b
Sep 10 17:36:49.281001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:49.280951    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls" event={"ID":"439ea1b8-0bc2-438b-a259-8ce664f5fc05","Type":"ContainerStarted","Data":"a9ab03ade42a362cb8392de35cff96e19411e0bbc428e92bd2a0ee208eff287c"}
Sep 10 17:36:49.281182 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:49.281030    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls" event={"ID":"439ea1b8-0bc2-438b-a259-8ce664f5fc05","Type":"ContainerStarted","Data":"c3a10b5a1bbb83225a6a7eb374485646d97b138ad74a0bb5c6357af18b2a3b1b"}
Sep 10 17:36:49.281287 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:49.281203    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:36:49.282443 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:49.282415    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z" event={"ID":"4f2b010b-fbd5-4059-a14d-0a5a52dd4354","Type":"ContainerStarted","Data":"044899384f00541699acabf1f0e82c39b6f2a7bae26bb19aad3f66fad13f969e"}
Sep 10 17:36:49.305849 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:49.305800    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls" podStartSLOduration=1.305786109 podStartE2EDuration="1.305786109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:49.303694874 +0000 UTC m=+778.013289141" watchObservedRunningTime="2026-09-10 17:36:49.305786109 +0000 UTC m=+778.015380375"
Sep 10 17:36:50.290396 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:50.290353    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z" event={"ID":"4f2b010b-fbd5-4059-a14d-0a5a52dd4354","Type":"ContainerStarted","Data":"bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4"}
Sep 10 17:36:50.290825 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:50.290431    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:36:50.320504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:50.320449    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z" podStartSLOduration=2.991156019 podStartE2EDuration="3.320433966s" totalImagesPullingTime="329.277947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:50.317692127 +0000 UTC m=+779.027286395" watchObservedRunningTime="2026-09-10 17:36:50.320433966 +0000 UTC m=+779.030028231"
Sep 10 17:36:56.301014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:36:56.300983    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:37:00.292405 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:00.292375    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:37:05.522170 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.522134    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"]
Sep 10 17:37:05.522983 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.522951    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n" podUID="2660991f-2a55-4149-aaa0-f8269c95486e" containerName="maas-api" containerID="cri-o://b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa" gracePeriod=30
Sep 10 17:37:05.583626 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.583592    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"]
Sep 10 17:37:05.583913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.583864    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n" podUID="68cf7190-b29a-4779-ae19-9ad4d48b9dba" containerName="payload-processing" containerID="cri-o://54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852" gracePeriod=30
Sep 10 17:37:05.589168 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.589140    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"]
Sep 10 17:37:05.590748 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.590510    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d" podUID="7330b0f2-07ee-4972-99eb-e248d850e7a3" containerName="payload-pre-processing" containerID="cri-o://af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407" gracePeriod=30
Sep 10 17:37:05.856417 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.856382    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:37:05.890011 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.889985    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:37:05.894450 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.894422    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-metrics-certs\") pod \"2660991f-2a55-4149-aaa0-f8269c95486e\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") "
Sep 10 17:37:05.894590 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.894522    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-maas-api-tls\") pod \"2660991f-2a55-4149-aaa0-f8269c95486e\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") "
Sep 10 17:37:05.894665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.894606    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb6rl\" (UniqueName: \"kubernetes.io/projected/2660991f-2a55-4149-aaa0-f8269c95486e-kube-api-access-gb6rl\") pod \"2660991f-2a55-4149-aaa0-f8269c95486e\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") "
Sep 10 17:37:05.894665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.894638    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660991f-2a55-4149-aaa0-f8269c95486e-openshift-service-ca\") pod \"2660991f-2a55-4149-aaa0-f8269c95486e\" (UID: \"2660991f-2a55-4149-aaa0-f8269c95486e\") "
Sep 10 17:37:05.895529 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.895498    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2660991f-2a55-4149-aaa0-f8269c95486e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2660991f-2a55-4149-aaa0-f8269c95486e" (UID: "2660991f-2a55-4149-aaa0-f8269c95486e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:05.899210 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.899173    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2660991f-2a55-4149-aaa0-f8269c95486e" (UID: "2660991f-2a55-4149-aaa0-f8269c95486e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:37:05.899948 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.899913    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2660991f-2a55-4149-aaa0-f8269c95486e" (UID: "2660991f-2a55-4149-aaa0-f8269c95486e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:37:05.900274 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.900217    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2660991f-2a55-4149-aaa0-f8269c95486e-kube-api-access-gb6rl" (OuterVolumeSpecName: "kube-api-access-gb6rl") pod "2660991f-2a55-4149-aaa0-f8269c95486e" (UID: "2660991f-2a55-4149-aaa0-f8269c95486e"). InnerVolumeSpecName "kube-api-access-gb6rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:05.908385 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.908364    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:37:05.995447 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995412    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5chb\" (UniqueName: \"kubernetes.io/projected/7330b0f2-07ee-4972-99eb-e248d850e7a3-kube-api-access-v5chb\") pod \"7330b0f2-07ee-4972-99eb-e248d850e7a3\" (UID: \"7330b0f2-07ee-4972-99eb-e248d850e7a3\") "
Sep 10 17:37:05.995447 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995446    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq2xk\" (UniqueName: \"kubernetes.io/projected/68cf7190-b29a-4779-ae19-9ad4d48b9dba-kube-api-access-cq2xk\") pod \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\" (UID: \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\") "
Sep 10 17:37:05.995688 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995466    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330b0f2-07ee-4972-99eb-e248d850e7a3-plugins-config-volume\") pod \"7330b0f2-07ee-4972-99eb-e248d850e7a3\" (UID: \"7330b0f2-07ee-4972-99eb-e248d850e7a3\") "
Sep 10 17:37:05.995688 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995488    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cf7190-b29a-4779-ae19-9ad4d48b9dba-plugins-config-volume\") pod \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\" (UID: \"68cf7190-b29a-4779-ae19-9ad4d48b9dba\") "
Sep 10 17:37:05.995806 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995752    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:05.995806 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995771    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660991f-2a55-4149-aaa0-f8269c95486e-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:05.995806 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995787    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb6rl\" (UniqueName: \"kubernetes.io/projected/2660991f-2a55-4149-aaa0-f8269c95486e-kube-api-access-gb6rl\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:05.995806 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995803    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660991f-2a55-4149-aaa0-f8269c95486e-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:05.995953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995887    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cf7190-b29a-4779-ae19-9ad4d48b9dba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68cf7190-b29a-4779-ae19-9ad4d48b9dba" (UID: "68cf7190-b29a-4779-ae19-9ad4d48b9dba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:05.995953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.995887    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7330b0f2-07ee-4972-99eb-e248d850e7a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7330b0f2-07ee-4972-99eb-e248d850e7a3" (UID: "7330b0f2-07ee-4972-99eb-e248d850e7a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:05.997632 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.997607    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7330b0f2-07ee-4972-99eb-e248d850e7a3-kube-api-access-v5chb" (OuterVolumeSpecName: "kube-api-access-v5chb") pod "7330b0f2-07ee-4972-99eb-e248d850e7a3" (UID: "7330b0f2-07ee-4972-99eb-e248d850e7a3"). InnerVolumeSpecName "kube-api-access-v5chb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:05.997706 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:05.997650    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cf7190-b29a-4779-ae19-9ad4d48b9dba-kube-api-access-cq2xk" (OuterVolumeSpecName: "kube-api-access-cq2xk") pod "68cf7190-b29a-4779-ae19-9ad4d48b9dba" (UID: "68cf7190-b29a-4779-ae19-9ad4d48b9dba"). InnerVolumeSpecName "kube-api-access-cq2xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:06.096908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.096827    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5chb\" (UniqueName: \"kubernetes.io/projected/7330b0f2-07ee-4972-99eb-e248d850e7a3-kube-api-access-v5chb\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:06.096908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.096856    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq2xk\" (UniqueName: \"kubernetes.io/projected/68cf7190-b29a-4779-ae19-9ad4d48b9dba-kube-api-access-cq2xk\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:06.096908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.096866    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330b0f2-07ee-4972-99eb-e248d850e7a3-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:06.096908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.096875    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cf7190-b29a-4779-ae19-9ad4d48b9dba-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:06.372829 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.372745    2568 generic.go:356] "Generic (PLEG): container finished" podID="68cf7190-b29a-4779-ae19-9ad4d48b9dba" containerID="54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852" exitCode=0
Sep 10 17:37:06.372829 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.372795    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"
Sep 10 17:37:06.373064 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.372830    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n" event={"ID":"68cf7190-b29a-4779-ae19-9ad4d48b9dba","Type":"ContainerDied","Data":"54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852"}
Sep 10 17:37:06.373064 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.372906    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n" event={"ID":"68cf7190-b29a-4779-ae19-9ad4d48b9dba","Type":"ContainerDied","Data":"1e7168ffc279f6271899671c64db22071231ceb5740cadd8750b9c1436d9cb94"}
Sep 10 17:37:06.373064 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.372937    2568 scope.go:117] "RemoveContainer" containerID="54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852"
Sep 10 17:37:06.374622 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.374570    2568 generic.go:356] "Generic (PLEG): container finished" podID="2660991f-2a55-4149-aaa0-f8269c95486e" containerID="b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa" exitCode=0
Sep 10 17:37:06.374751 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.374694    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n" event={"ID":"2660991f-2a55-4149-aaa0-f8269c95486e","Type":"ContainerDied","Data":"b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa"}
Sep 10 17:37:06.374751 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.374744    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n" event={"ID":"2660991f-2a55-4149-aaa0-f8269c95486e","Type":"ContainerDied","Data":"b74af79b8f21e134c98454ce36c52f9f910584e2179f1e3bd7948868f9ab4742"}
Sep 10 17:37:06.374906 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.374818    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"
Sep 10 17:37:06.376347 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.376323    2568 generic.go:356] "Generic (PLEG): container finished" podID="7330b0f2-07ee-4972-99eb-e248d850e7a3" containerID="af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407" exitCode=0
Sep 10 17:37:06.376443 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.376356    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"
Sep 10 17:37:06.376443 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.376398    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d" event={"ID":"7330b0f2-07ee-4972-99eb-e248d850e7a3","Type":"ContainerDied","Data":"af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407"}
Sep 10 17:37:06.376443 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.376439    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d" event={"ID":"7330b0f2-07ee-4972-99eb-e248d850e7a3","Type":"ContainerDied","Data":"2b117715c410cf8cab7e6e1fa20cb5f113cf8942538f238ed446092bf5352775"}
Sep 10 17:37:06.385634 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.385617    2568 scope.go:117] "RemoveContainer" containerID="54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852"
Sep 10 17:37:06.385896 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:37:06.385876    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852\": container with ID starting with 54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852 not found: ID does not exist" containerID="54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852"
Sep 10 17:37:06.385953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.385905    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852"} err="failed to get container status \"54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852\": rpc error: code = NotFound desc = could not find container \"54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852\": container with ID starting with 54d4c16699cad36762e470a362635ba132e56eafaa2a77e919e1534384cc3852 not found: ID does not exist"
Sep 10 17:37:06.385953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.385921    2568 scope.go:117] "RemoveContainer" containerID="b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa"
Sep 10 17:37:06.394103 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.394086    2568 scope.go:117] "RemoveContainer" containerID="b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa"
Sep 10 17:37:06.394408 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:37:06.394382    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa\": container with ID starting with b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa not found: ID does not exist" containerID="b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa"
Sep 10 17:37:06.394477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.394417    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa"} err="failed to get container status \"b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa\": rpc error: code = NotFound desc = could not find container \"b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa\": container with ID starting with b9a6a232782b3f91bc01c1de3f18fa5d35b163f6d2d0f0b8b0a4c26f031522aa not found: ID does not exist"
Sep 10 17:37:06.394477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.394434    2568 scope.go:117] "RemoveContainer" containerID="af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407"
Sep 10 17:37:06.401642 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.401624    2568 scope.go:117] "RemoveContainer" containerID="af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407"
Sep 10 17:37:06.401903 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:37:06.401886    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407\": container with ID starting with af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407 not found: ID does not exist" containerID="af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407"
Sep 10 17:37:06.401944 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.401910    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407"} err="failed to get container status \"af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407\": rpc error: code = NotFound desc = could not find container \"af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407\": container with ID starting with af77b6e259fb9804cb70195201ed437db64a7092bdf6dbda81827cfd19fd8407 not found: ID does not exist"
Sep 10 17:37:06.420067 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.420042    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"]
Sep 10 17:37:06.430386 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.430365    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64df4d9a-8559c8b7f9-xd72n"]
Sep 10 17:37:06.446712 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.446691    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"]
Sep 10 17:37:06.457472 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.457445    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-64df4d9a-84d645d956-75d6n"]
Sep 10 17:37:06.499875 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.499849    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"]
Sep 10 17:37:06.507056 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:06.507034    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64df4d9a-5b79cc988b-q6h5d"]
Sep 10 17:37:07.764050 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:07.764017    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2660991f-2a55-4149-aaa0-f8269c95486e" path="/var/lib/kubelet/pods/2660991f-2a55-4149-aaa0-f8269c95486e/volumes"
Sep 10 17:37:07.764431 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:07.764407    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68cf7190-b29a-4779-ae19-9ad4d48b9dba" path="/var/lib/kubelet/pods/68cf7190-b29a-4779-ae19-9ad4d48b9dba/volumes"
Sep 10 17:37:07.764728 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:07.764715    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7330b0f2-07ee-4972-99eb-e248d850e7a3" path="/var/lib/kubelet/pods/7330b0f2-07ee-4972-99eb-e248d850e7a3/volumes"
Sep 10 17:37:09.740268 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.740215    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"]
Sep 10 17:37:09.740838 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.740814    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68cf7190-b29a-4779-ae19-9ad4d48b9dba" containerName="payload-processing"
Sep 10 17:37:09.740927 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.740842    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="68cf7190-b29a-4779-ae19-9ad4d48b9dba" containerName="payload-processing"
Sep 10 17:37:09.740927 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.740858    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7330b0f2-07ee-4972-99eb-e248d850e7a3" containerName="payload-pre-processing"
Sep 10 17:37:09.740927 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.740866    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7330b0f2-07ee-4972-99eb-e248d850e7a3" containerName="payload-pre-processing"
Sep 10 17:37:09.740927 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.740899    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2660991f-2a55-4149-aaa0-f8269c95486e" containerName="maas-api"
Sep 10 17:37:09.740927 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.740908    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2660991f-2a55-4149-aaa0-f8269c95486e" containerName="maas-api"
Sep 10 17:37:09.741201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.741028    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2660991f-2a55-4149-aaa0-f8269c95486e" containerName="maas-api"
Sep 10 17:37:09.741201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.741042    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="68cf7190-b29a-4779-ae19-9ad4d48b9dba" containerName="payload-processing"
Sep 10 17:37:09.741201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.741058    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7330b0f2-07ee-4972-99eb-e248d850e7a3" containerName="payload-pre-processing"
Sep 10 17:37:09.744652 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.744628    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.748142 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.748118    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fb3998b8-serving-cert\""
Sep 10 17:37:09.756883 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.756859    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"]
Sep 10 17:37:09.832003 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.831966    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-maas-api-tls\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.832003 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.832003    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-metrics-certs\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.832211 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.832048    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7333a88c-91e2-40ca-ab0f-f51323bf680b-openshift-service-ca\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.832211 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.832091    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzk8\" (UniqueName: \"kubernetes.io/projected/7333a88c-91e2-40ca-ab0f-f51323bf680b-kube-api-access-gwzk8\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.933013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.932974    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-maas-api-tls\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.933013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.933012    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-metrics-certs\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.933334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.933146    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7333a88c-91e2-40ca-ab0f-f51323bf680b-openshift-service-ca\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.933334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.933223    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzk8\" (UniqueName: \"kubernetes.io/projected/7333a88c-91e2-40ca-ab0f-f51323bf680b-kube-api-access-gwzk8\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.933788 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.933765    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7333a88c-91e2-40ca-ab0f-f51323bf680b-openshift-service-ca\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.935487 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.935465    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-maas-api-tls\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.935580 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.935504    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-metrics-certs\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:09.942441 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:09.942416    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzk8\" (UniqueName: \"kubernetes.io/projected/7333a88c-91e2-40ca-ab0f-f51323bf680b-kube-api-access-gwzk8\") pod \"maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:10.057572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.057532    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:10.363822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.363724    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"]
Sep 10 17:37:10.367385 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:37:10.367359    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7333a88c_91e2_40ca_ab0f_f51323bf680b.slice/crio-f7c923f3d74110940ea22d7e549d03549652cf69cddef17590493ba9ece9544f WatchSource:0}: Error finding container f7c923f3d74110940ea22d7e549d03549652cf69cddef17590493ba9ece9544f: Status 404 returned error can't find the container with id f7c923f3d74110940ea22d7e549d03549652cf69cddef17590493ba9ece9544f
Sep 10 17:37:10.400486 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.400450    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl" event={"ID":"7333a88c-91e2-40ca-ab0f-f51323bf680b","Type":"ContainerStarted","Data":"f7c923f3d74110940ea22d7e549d03549652cf69cddef17590493ba9ece9544f"}
Sep 10 17:37:10.506094 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.506051    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"]
Sep 10 17:37:10.509667 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.509637    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:10.510938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.510907    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"]
Sep 10 17:37:10.512817 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.512792    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fb3998b8\""
Sep 10 17:37:10.512940 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.512834    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fb3998b8-dockercfg-8w9hs\""
Sep 10 17:37:10.513651 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.513632    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:10.522910 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.522876    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"]
Sep 10 17:37:10.529661 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.529637    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"]
Sep 10 17:37:10.641666 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.641568    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc40ac69-8059-44ed-989f-a85d48727121-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh\" (UID: \"fc40ac69-8059-44ed-989f-a85d48727121\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:10.641666 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.641640    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6f9\" (UniqueName: \"kubernetes.io/projected/fc40ac69-8059-44ed-989f-a85d48727121-kube-api-access-pq6f9\") pod \"payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh\" (UID: \"fc40ac69-8059-44ed-989f-a85d48727121\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:10.641920 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.641781    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zdp\" (UniqueName: \"kubernetes.io/projected/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-kube-api-access-w6zdp\") pod \"payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz\" (UID: \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:10.641920 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.641834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz\" (UID: \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:10.742999 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.742968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz\" (UID: \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:10.743420 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.743026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc40ac69-8059-44ed-989f-a85d48727121-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh\" (UID: \"fc40ac69-8059-44ed-989f-a85d48727121\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:10.743420 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.743065    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6f9\" (UniqueName: \"kubernetes.io/projected/fc40ac69-8059-44ed-989f-a85d48727121-kube-api-access-pq6f9\") pod \"payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh\" (UID: \"fc40ac69-8059-44ed-989f-a85d48727121\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:10.743420 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.743170    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zdp\" (UniqueName: \"kubernetes.io/projected/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-kube-api-access-w6zdp\") pod \"payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz\" (UID: \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:10.743877 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.743854    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc40ac69-8059-44ed-989f-a85d48727121-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh\" (UID: \"fc40ac69-8059-44ed-989f-a85d48727121\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:10.743956 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.743892    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz\" (UID: \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:10.756127 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.756101    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6f9\" (UniqueName: \"kubernetes.io/projected/fc40ac69-8059-44ed-989f-a85d48727121-kube-api-access-pq6f9\") pod \"payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh\" (UID: \"fc40ac69-8059-44ed-989f-a85d48727121\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:10.756214 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.756138    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zdp\" (UniqueName: \"kubernetes.io/projected/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-kube-api-access-w6zdp\") pod \"payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz\" (UID: \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:10.823000 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.822964    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:10.832133 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:10.832112    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:11.104335 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.104305    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"]
Sep 10 17:37:11.106595 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:37:11.106567    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dac70ce_b437_4ed9_9c9b_b4ac43427c12.slice/crio-0a3f54a07fabfe3bf3ad139543c971eddec80fa56e8c19acd09df0868223b9eb WatchSource:0}: Error finding container 0a3f54a07fabfe3bf3ad139543c971eddec80fa56e8c19acd09df0868223b9eb: Status 404 returned error can't find the container with id 0a3f54a07fabfe3bf3ad139543c971eddec80fa56e8c19acd09df0868223b9eb
Sep 10 17:37:11.123085 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.123056    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"]
Sep 10 17:37:11.123946 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:37:11.123918    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc40ac69_8059_44ed_989f_a85d48727121.slice/crio-7219db1a1bdbdda7f350016051a4a72249d616687a3c6383dc3efb1fa5e7ad9c WatchSource:0}: Error finding container 7219db1a1bdbdda7f350016051a4a72249d616687a3c6383dc3efb1fa5e7ad9c: Status 404 returned error can't find the container with id 7219db1a1bdbdda7f350016051a4a72249d616687a3c6383dc3efb1fa5e7ad9c
Sep 10 17:37:11.414258 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.414128    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl" event={"ID":"7333a88c-91e2-40ca-ab0f-f51323bf680b","Type":"ContainerStarted","Data":"42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842"}
Sep 10 17:37:11.414462 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.414263    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:11.415820 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.415792    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh" event={"ID":"fc40ac69-8059-44ed-989f-a85d48727121","Type":"ContainerStarted","Data":"5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd"}
Sep 10 17:37:11.415923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.415842    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh" event={"ID":"fc40ac69-8059-44ed-989f-a85d48727121","Type":"ContainerStarted","Data":"7219db1a1bdbdda7f350016051a4a72249d616687a3c6383dc3efb1fa5e7ad9c"}
Sep 10 17:37:11.415923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.415865    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:11.417502 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.417482    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz" event={"ID":"2dac70ce-b437-4ed9-9c9b-b4ac43427c12","Type":"ContainerStarted","Data":"6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4"}
Sep 10 17:37:11.417605 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.417524    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz" event={"ID":"2dac70ce-b437-4ed9-9c9b-b4ac43427c12","Type":"ContainerStarted","Data":"0a3f54a07fabfe3bf3ad139543c971eddec80fa56e8c19acd09df0868223b9eb"}
Sep 10 17:37:11.417605 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.417575    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:11.432662 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.432618    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl" podStartSLOduration=2.076950418 podStartE2EDuration="2.432603797s" totalImagesPullingTime="355.653379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:11.432373586 +0000 UTC m=+800.141967856" watchObservedRunningTime="2026-09-10 17:37:11.432603797 +0000 UTC m=+800.142198064"
Sep 10 17:37:11.450162 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.450106    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz" podStartSLOduration=1.450091182 podStartE2EDuration="1.450091182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:11.448961457 +0000 UTC m=+800.158555724" watchObservedRunningTime="2026-09-10 17:37:11.450091182 +0000 UTC m=+800.159685448"
Sep 10 17:37:11.468986 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:11.468913    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh" podStartSLOduration=1.468891012 podStartE2EDuration="1.468891012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:11.468612979 +0000 UTC m=+800.178207246" watchObservedRunningTime="2026-09-10 17:37:11.468891012 +0000 UTC m=+800.178485280"
Sep 10 17:37:17.429277 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:17.429172    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:22.424917 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:22.424885    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:22.425433 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:22.424978    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:40.689274 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:40.689200    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"]
Sep 10 17:37:40.689781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:40.689480    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl" podUID="7333a88c-91e2-40ca-ab0f-f51323bf680b" containerName="maas-api" containerID="cri-o://42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842" gracePeriod=30
Sep 10 17:37:40.734593 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:40.734563    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"]
Sep 10 17:37:40.734843 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:40.734784    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh" podUID="fc40ac69-8059-44ed-989f-a85d48727121" containerName="payload-processing" containerID="cri-o://5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd" gracePeriod=30
Sep 10 17:37:40.742582 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:40.742381    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"]
Sep 10 17:37:40.742719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:40.742577    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz" podUID="2dac70ce-b437-4ed9-9c9b-b4ac43427c12" containerName="payload-pre-processing" containerID="cri-o://6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4" gracePeriod=30
Sep 10 17:37:41.016461 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.016356    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:41.026710 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.026575    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-metrics-certs\") pod \"7333a88c-91e2-40ca-ab0f-f51323bf680b\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") "
Sep 10 17:37:41.027252 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.026682    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7333a88c-91e2-40ca-ab0f-f51323bf680b-openshift-service-ca\") pod \"7333a88c-91e2-40ca-ab0f-f51323bf680b\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") "
Sep 10 17:37:41.027252 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.026935    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-maas-api-tls\") pod \"7333a88c-91e2-40ca-ab0f-f51323bf680b\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") "
Sep 10 17:37:41.027252 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.026971    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwzk8\" (UniqueName: \"kubernetes.io/projected/7333a88c-91e2-40ca-ab0f-f51323bf680b-kube-api-access-gwzk8\") pod \"7333a88c-91e2-40ca-ab0f-f51323bf680b\" (UID: \"7333a88c-91e2-40ca-ab0f-f51323bf680b\") "
Sep 10 17:37:41.033382 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.030998    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7333a88c-91e2-40ca-ab0f-f51323bf680b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7333a88c-91e2-40ca-ab0f-f51323bf680b" (UID: "7333a88c-91e2-40ca-ab0f-f51323bf680b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:41.036189 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.036052    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7333a88c-91e2-40ca-ab0f-f51323bf680b" (UID: "7333a88c-91e2-40ca-ab0f-f51323bf680b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:37:41.036189 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.036155    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7333a88c-91e2-40ca-ab0f-f51323bf680b" (UID: "7333a88c-91e2-40ca-ab0f-f51323bf680b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:37:41.036363 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.036225    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7333a88c-91e2-40ca-ab0f-f51323bf680b-kube-api-access-gwzk8" (OuterVolumeSpecName: "kube-api-access-gwzk8") pod "7333a88c-91e2-40ca-ab0f-f51323bf680b" (UID: "7333a88c-91e2-40ca-ab0f-f51323bf680b"). InnerVolumeSpecName "kube-api-access-gwzk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:41.065251 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.065213    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:41.068991 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.068972    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:41.131481 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131453    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc40ac69-8059-44ed-989f-a85d48727121-plugins-config-volume\") pod \"fc40ac69-8059-44ed-989f-a85d48727121\" (UID: \"fc40ac69-8059-44ed-989f-a85d48727121\") "
Sep 10 17:37:41.131645 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131504    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq6f9\" (UniqueName: \"kubernetes.io/projected/fc40ac69-8059-44ed-989f-a85d48727121-kube-api-access-pq6f9\") pod \"fc40ac69-8059-44ed-989f-a85d48727121\" (UID: \"fc40ac69-8059-44ed-989f-a85d48727121\") "
Sep 10 17:37:41.131645 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131537    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-plugins-config-volume\") pod \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\" (UID: \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\") "
Sep 10 17:37:41.131645 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131577    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6zdp\" (UniqueName: \"kubernetes.io/projected/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-kube-api-access-w6zdp\") pod \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\" (UID: \"2dac70ce-b437-4ed9-9c9b-b4ac43427c12\") "
Sep 10 17:37:41.131804 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131770    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7333a88c-91e2-40ca-ab0f-f51323bf680b-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:41.131804 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131782    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:41.131804 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131792    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwzk8\" (UniqueName: \"kubernetes.io/projected/7333a88c-91e2-40ca-ab0f-f51323bf680b-kube-api-access-gwzk8\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:41.131804 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131801    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7333a88c-91e2-40ca-ab0f-f51323bf680b-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:41.132001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131889    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc40ac69-8059-44ed-989f-a85d48727121-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc40ac69-8059-44ed-989f-a85d48727121" (UID: "fc40ac69-8059-44ed-989f-a85d48727121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:41.132001 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.131906    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dac70ce-b437-4ed9-9c9b-b4ac43427c12" (UID: "2dac70ce-b437-4ed9-9c9b-b4ac43427c12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:41.133639 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.133618    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-kube-api-access-w6zdp" (OuterVolumeSpecName: "kube-api-access-w6zdp") pod "2dac70ce-b437-4ed9-9c9b-b4ac43427c12" (UID: "2dac70ce-b437-4ed9-9c9b-b4ac43427c12"). InnerVolumeSpecName "kube-api-access-w6zdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:41.133721 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.133649    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc40ac69-8059-44ed-989f-a85d48727121-kube-api-access-pq6f9" (OuterVolumeSpecName: "kube-api-access-pq6f9") pod "fc40ac69-8059-44ed-989f-a85d48727121" (UID: "fc40ac69-8059-44ed-989f-a85d48727121"). InnerVolumeSpecName "kube-api-access-pq6f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:41.232508 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.232432    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc40ac69-8059-44ed-989f-a85d48727121-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:41.232508 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.232454    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq6f9\" (UniqueName: \"kubernetes.io/projected/fc40ac69-8059-44ed-989f-a85d48727121-kube-api-access-pq6f9\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:41.232508 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.232466    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:41.232508 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.232476    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zdp\" (UniqueName: \"kubernetes.io/projected/2dac70ce-b437-4ed9-9c9b-b4ac43427c12-kube-api-access-w6zdp\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:37:41.589429 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.589389    2568 generic.go:356] "Generic (PLEG): container finished" podID="7333a88c-91e2-40ca-ab0f-f51323bf680b" containerID="42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842" exitCode=0
Sep 10 17:37:41.589628 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.589436    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"
Sep 10 17:37:41.589628 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.589465    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl" event={"ID":"7333a88c-91e2-40ca-ab0f-f51323bf680b","Type":"ContainerDied","Data":"42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842"}
Sep 10 17:37:41.589628 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.589530    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl" event={"ID":"7333a88c-91e2-40ca-ab0f-f51323bf680b","Type":"ContainerDied","Data":"f7c923f3d74110940ea22d7e549d03549652cf69cddef17590493ba9ece9544f"}
Sep 10 17:37:41.589628 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.589556    2568 scope.go:117] "RemoveContainer" containerID="42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842"
Sep 10 17:37:41.591461 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.591435    2568 generic.go:356] "Generic (PLEG): container finished" podID="fc40ac69-8059-44ed-989f-a85d48727121" containerID="5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd" exitCode=0
Sep 10 17:37:41.591607 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.591499    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"
Sep 10 17:37:41.591607 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.591524    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh" event={"ID":"fc40ac69-8059-44ed-989f-a85d48727121","Type":"ContainerDied","Data":"5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd"}
Sep 10 17:37:41.591607 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.591584    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh" event={"ID":"fc40ac69-8059-44ed-989f-a85d48727121","Type":"ContainerDied","Data":"7219db1a1bdbdda7f350016051a4a72249d616687a3c6383dc3efb1fa5e7ad9c"}
Sep 10 17:37:41.593348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.593261    2568 generic.go:356] "Generic (PLEG): container finished" podID="2dac70ce-b437-4ed9-9c9b-b4ac43427c12" containerID="6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4" exitCode=0
Sep 10 17:37:41.593348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.593320    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz" event={"ID":"2dac70ce-b437-4ed9-9c9b-b4ac43427c12","Type":"ContainerDied","Data":"6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4"}
Sep 10 17:37:41.593596 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.593359    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz" event={"ID":"2dac70ce-b437-4ed9-9c9b-b4ac43427c12","Type":"ContainerDied","Data":"0a3f54a07fabfe3bf3ad139543c971eddec80fa56e8c19acd09df0868223b9eb"}
Sep 10 17:37:41.593596 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.593421    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"
Sep 10 17:37:41.601704 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.601555    2568 scope.go:117] "RemoveContainer" containerID="42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842"
Sep 10 17:37:41.601971 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:37:41.601839    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842\": container with ID starting with 42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842 not found: ID does not exist" containerID="42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842"
Sep 10 17:37:41.601971 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.601875    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842"} err="failed to get container status \"42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842\": rpc error: code = NotFound desc = could not find container \"42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842\": container with ID starting with 42fefac255726a31f828fc5683f160b88f0a466b4466ce128277d117d1f37842 not found: ID does not exist"
Sep 10 17:37:41.601971 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.601897    2568 scope.go:117] "RemoveContainer" containerID="5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd"
Sep 10 17:37:41.610577 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.610561    2568 scope.go:117] "RemoveContainer" containerID="5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd"
Sep 10 17:37:41.610849 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:37:41.610826    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd\": container with ID starting with 5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd not found: ID does not exist" containerID="5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd"
Sep 10 17:37:41.610906 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.610858    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd"} err="failed to get container status \"5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd\": rpc error: code = NotFound desc = could not find container \"5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd\": container with ID starting with 5bd9075b28a1742d54fb18258b6c66888231ec10a479e3d7e457c3d04da66dbd not found: ID does not exist"
Sep 10 17:37:41.610906 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.610874    2568 scope.go:117] "RemoveContainer" containerID="6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4"
Sep 10 17:37:41.620150 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.620133    2568 scope.go:117] "RemoveContainer" containerID="6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4"
Sep 10 17:37:41.620454 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:37:41.620430    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4\": container with ID starting with 6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4 not found: ID does not exist" containerID="6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4"
Sep 10 17:37:41.620548 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.620461    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4"} err="failed to get container status \"6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4\": rpc error: code = NotFound desc = could not find container \"6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4\": container with ID starting with 6d208623b5fa9727047140c7ad87d49b6e3cee5e3469926273911b46651836a4 not found: ID does not exist"
Sep 10 17:37:41.620710 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.620693    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"]
Sep 10 17:37:41.628681 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.628662    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb3998b8-6fd675d774-grmcl"]
Sep 10 17:37:41.640444 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.640410    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"]
Sep 10 17:37:41.649949 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.649925    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb3998b8-b4d4dbd5d-x4zdh"]
Sep 10 17:37:41.661314 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.661289    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"]
Sep 10 17:37:41.666011 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.665992    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb3998b8-5f8cccc6b-7wjbz"]
Sep 10 17:37:41.765386 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.765353    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dac70ce-b437-4ed9-9c9b-b4ac43427c12" path="/var/lib/kubelet/pods/2dac70ce-b437-4ed9-9c9b-b4ac43427c12/volumes"
Sep 10 17:37:41.765926 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.765904    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7333a88c-91e2-40ca-ab0f-f51323bf680b" path="/var/lib/kubelet/pods/7333a88c-91e2-40ca-ab0f-f51323bf680b/volumes"
Sep 10 17:37:41.766286 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:41.766270    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc40ac69-8059-44ed-989f-a85d48727121" path="/var/lib/kubelet/pods/fc40ac69-8059-44ed-989f-a85d48727121/volumes"
Sep 10 17:37:45.967340 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.967296    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"]
Sep 10 17:37:45.967968 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.967935    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc40ac69-8059-44ed-989f-a85d48727121" containerName="payload-processing"
Sep 10 17:37:45.967968 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.967963    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc40ac69-8059-44ed-989f-a85d48727121" containerName="payload-processing"
Sep 10 17:37:45.968121 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.967997    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7333a88c-91e2-40ca-ab0f-f51323bf680b" containerName="maas-api"
Sep 10 17:37:45.968121 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.968006    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7333a88c-91e2-40ca-ab0f-f51323bf680b" containerName="maas-api"
Sep 10 17:37:45.968121 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.968025    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac70ce-b437-4ed9-9c9b-b4ac43427c12" containerName="payload-pre-processing"
Sep 10 17:37:45.968121 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.968033    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac70ce-b437-4ed9-9c9b-b4ac43427c12" containerName="payload-pre-processing"
Sep 10 17:37:45.968358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.968134    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc40ac69-8059-44ed-989f-a85d48727121" containerName="payload-processing"
Sep 10 17:37:45.968358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.968147    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac70ce-b437-4ed9-9c9b-b4ac43427c12" containerName="payload-pre-processing"
Sep 10 17:37:45.968358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.968161    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7333a88c-91e2-40ca-ab0f-f51323bf680b" containerName="maas-api"
Sep 10 17:37:45.973190 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.973166    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:45.975772 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.975749    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b0b7bb86-serving-cert\""
Sep 10 17:37:45.982678 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:45.982653    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"]
Sep 10 17:37:46.076107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.076070    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-metrics-certs\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.076301 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.076129    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pkg\" (UniqueName: \"kubernetes.io/projected/d38f61de-7617-4e38-ab7c-47b7a68f7c26-kube-api-access-t5pkg\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.076301 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.076179    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38f61de-7617-4e38-ab7c-47b7a68f7c26-openshift-service-ca\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.076301 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.076258    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.177170 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.177133    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.177170 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.177172    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-metrics-certs\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.177459 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.177219    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pkg\" (UniqueName: \"kubernetes.io/projected/d38f61de-7617-4e38-ab7c-47b7a68f7c26-kube-api-access-t5pkg\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.177459 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.177306    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38f61de-7617-4e38-ab7c-47b7a68f7c26-openshift-service-ca\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.177459 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:37:46.177306    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-serving-cert: secret "maas-api-e2e-ait-b0b7bb86-serving-cert" not found
Sep 10 17:37:46.177459 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:37:46.177405    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls podName:d38f61de-7617-4e38-ab7c-47b7a68f7c26 nodeName:}" failed. No retries permitted until 2026-09-10 17:37:46.677385912 +0000 UTC m=+835.386980156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls") pod "maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq" (UID: "d38f61de-7617-4e38-ab7c-47b7a68f7c26") : secret "maas-api-e2e-ait-b0b7bb86-serving-cert" not found
Sep 10 17:37:46.178046 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.178021    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38f61de-7617-4e38-ab7c-47b7a68f7c26-openshift-service-ca\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.179796 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.179774    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-metrics-certs\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.189203 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.189172    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pkg\" (UniqueName: \"kubernetes.io/projected/d38f61de-7617-4e38-ab7c-47b7a68f7c26-kube-api-access-t5pkg\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.587285 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.587247    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"]
Sep 10 17:37:46.591002 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.590984    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:46.593924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.593903    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b0b7bb86\""
Sep 10 17:37:46.594128 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.594113    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b0b7bb86-dockercfg-6pjwz\""
Sep 10 17:37:46.602563 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.602542    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"]
Sep 10 17:37:46.614985 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.614963    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"]
Sep 10 17:37:46.620445 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.620415    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:46.632194 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.632168    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"]
Sep 10 17:37:46.682605 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.682572    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k\" (UID: \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:46.682766 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.682634    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.682766 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.682692    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79636d0d-de6b-441f-a454-97bb355c6bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7\" (UID: \"79636d0d-de6b-441f-a454-97bb355c6bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:46.682766 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.682724    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr4gj\" (UniqueName: \"kubernetes.io/projected/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-kube-api-access-hr4gj\") pod \"payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k\" (UID: \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:46.682889 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.682794    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ttl\" (UniqueName: \"kubernetes.io/projected/79636d0d-de6b-441f-a454-97bb355c6bde-kube-api-access-h7ttl\") pod \"payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7\" (UID: \"79636d0d-de6b-441f-a454-97bb355c6bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:46.685024 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.684997    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls\") pod \"maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.784193 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.784144    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79636d0d-de6b-441f-a454-97bb355c6bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7\" (UID: \"79636d0d-de6b-441f-a454-97bb355c6bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:46.784193 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.784195    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr4gj\" (UniqueName: \"kubernetes.io/projected/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-kube-api-access-hr4gj\") pod \"payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k\" (UID: \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:46.784477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.784262    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ttl\" (UniqueName: \"kubernetes.io/projected/79636d0d-de6b-441f-a454-97bb355c6bde-kube-api-access-h7ttl\") pod \"payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7\" (UID: \"79636d0d-de6b-441f-a454-97bb355c6bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:46.784477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.784355    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k\" (UID: \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:46.785152 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.785123    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79636d0d-de6b-441f-a454-97bb355c6bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7\" (UID: \"79636d0d-de6b-441f-a454-97bb355c6bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:46.785316 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.785295    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k\" (UID: \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:46.792990 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.792967    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ttl\" (UniqueName: \"kubernetes.io/projected/79636d0d-de6b-441f-a454-97bb355c6bde-kube-api-access-h7ttl\") pod \"payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7\" (UID: \"79636d0d-de6b-441f-a454-97bb355c6bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:46.793090 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.793059    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr4gj\" (UniqueName: \"kubernetes.io/projected/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-kube-api-access-hr4gj\") pod \"payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k\" (UID: \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:46.885617 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.885537    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:46.903360 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.903333    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:46.934250 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:46.934196    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:47.161781 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.161751    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"]
Sep 10 17:37:47.162931 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:37:47.162906    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38f61de_7617_4e38_ab7c_47b7a68f7c26.slice/crio-a3b585b922eb161fe9b87b1de883e2769f5d3fe57cef2742325c76fee6c7ed17 WatchSource:0}: Error finding container a3b585b922eb161fe9b87b1de883e2769f5d3fe57cef2742325c76fee6c7ed17: Status 404 returned error can't find the container with id a3b585b922eb161fe9b87b1de883e2769f5d3fe57cef2742325c76fee6c7ed17
Sep 10 17:37:47.193716 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.193693    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"]
Sep 10 17:37:47.195652 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:37:47.195625    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79636d0d_de6b_441f_a454_97bb355c6bde.slice/crio-57f2ca24e90f94c5b640abd6269a4a65bff2ae4ef1af1c1e87f61d683ecc35e6 WatchSource:0}: Error finding container 57f2ca24e90f94c5b640abd6269a4a65bff2ae4ef1af1c1e87f61d683ecc35e6: Status 404 returned error can't find the container with id 57f2ca24e90f94c5b640abd6269a4a65bff2ae4ef1af1c1e87f61d683ecc35e6
Sep 10 17:37:47.231953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.231910    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"]
Sep 10 17:37:47.233463 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:37:47.233436    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020e9b23_ccf5_4181_a6c9_9e2f22b61cdb.slice/crio-860e58293c6ab4000ace6a7a08af54588451d23791dc8c47332dfb4789008dd1 WatchSource:0}: Error finding container 860e58293c6ab4000ace6a7a08af54588451d23791dc8c47332dfb4789008dd1: Status 404 returned error can't find the container with id 860e58293c6ab4000ace6a7a08af54588451d23791dc8c47332dfb4789008dd1
Sep 10 17:37:47.637099 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.637066    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k" event={"ID":"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb","Type":"ContainerStarted","Data":"e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7"}
Sep 10 17:37:47.637315 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.637124    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k" event={"ID":"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb","Type":"ContainerStarted","Data":"860e58293c6ab4000ace6a7a08af54588451d23791dc8c47332dfb4789008dd1"}
Sep 10 17:37:47.637315 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.637183    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:47.638963 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.638936    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7" event={"ID":"79636d0d-de6b-441f-a454-97bb355c6bde","Type":"ContainerStarted","Data":"6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3"}
Sep 10 17:37:47.639079 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.638984    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7" event={"ID":"79636d0d-de6b-441f-a454-97bb355c6bde","Type":"ContainerStarted","Data":"57f2ca24e90f94c5b640abd6269a4a65bff2ae4ef1af1c1e87f61d683ecc35e6"}
Sep 10 17:37:47.639079 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.639025    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:37:47.640521 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.640495    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq" event={"ID":"d38f61de-7617-4e38-ab7c-47b7a68f7c26","Type":"ContainerStarted","Data":"440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913"}
Sep 10 17:37:47.640642 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.640540    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq" event={"ID":"d38f61de-7617-4e38-ab7c-47b7a68f7c26","Type":"ContainerStarted","Data":"a3b585b922eb161fe9b87b1de883e2769f5d3fe57cef2742325c76fee6c7ed17"}
Sep 10 17:37:47.640642 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.640578    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:47.659733 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.659686    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k" podStartSLOduration=1.659668832 podStartE2EDuration="1.659668832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:47.656545355 +0000 UTC m=+836.366139648" watchObservedRunningTime="2026-09-10 17:37:47.659668832 +0000 UTC m=+836.369263105"
Sep 10 17:37:47.675810 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.675764    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq" podStartSLOduration=2.343687803 podStartE2EDuration="2.675751199s" totalImagesPullingTime="332.063396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:47.674732798 +0000 UTC m=+836.384327069" watchObservedRunningTime="2026-09-10 17:37:47.675751199 +0000 UTC m=+836.385345496"
Sep 10 17:37:47.692884 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:47.692843    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7" podStartSLOduration=1.6928319680000001 podStartE2EDuration="1.692831968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:47.691348991 +0000 UTC m=+836.400943265" watchObservedRunningTime="2026-09-10 17:37:47.692831968 +0000 UTC m=+836.402426234"
Sep 10 17:37:53.654938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:53.654909    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:37:58.651579 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:58.651550    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:37:58.651947 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:37:58.651616    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:38:16.397640 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.397605    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"]
Sep 10 17:38:16.398218 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.397862    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq" podUID="d38f61de-7617-4e38-ab7c-47b7a68f7c26" containerName="maas-api" containerID="cri-o://440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913" gracePeriod=30
Sep 10 17:38:16.451463 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.451426    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"]
Sep 10 17:38:16.451693 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.451664    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k" podUID="020e9b23-ccf5-4181-a6c9-9e2f22b61cdb" containerName="payload-processing" containerID="cri-o://e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7" gracePeriod=30
Sep 10 17:38:16.456857 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.456687    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"]
Sep 10 17:38:16.456979 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.456887    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7" podUID="79636d0d-de6b-441f-a454-97bb355c6bde" containerName="payload-pre-processing" containerID="cri-o://6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3" gracePeriod=30
Sep 10 17:38:16.719755 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.719731    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:38:16.754493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.754338    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:38:16.761603 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.761579    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:38:16.796361 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.796330    2568 generic.go:356] "Generic (PLEG): container finished" podID="d38f61de-7617-4e38-ab7c-47b7a68f7c26" containerID="440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913" exitCode=0
Sep 10 17:38:16.796509 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.796370    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"
Sep 10 17:38:16.796509 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.796413    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq" event={"ID":"d38f61de-7617-4e38-ab7c-47b7a68f7c26","Type":"ContainerDied","Data":"440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913"}
Sep 10 17:38:16.796509 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.796470    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq" event={"ID":"d38f61de-7617-4e38-ab7c-47b7a68f7c26","Type":"ContainerDied","Data":"a3b585b922eb161fe9b87b1de883e2769f5d3fe57cef2742325c76fee6c7ed17"}
Sep 10 17:38:16.796509 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.796506    2568 scope.go:117] "RemoveContainer" containerID="440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913"
Sep 10 17:38:16.797772 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.797750    2568 generic.go:356] "Generic (PLEG): container finished" podID="020e9b23-ccf5-4181-a6c9-9e2f22b61cdb" containerID="e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7" exitCode=0
Sep 10 17:38:16.797906 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.797789    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"
Sep 10 17:38:16.797906 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.797816    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k" event={"ID":"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb","Type":"ContainerDied","Data":"e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7"}
Sep 10 17:38:16.797906 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.797851    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k" event={"ID":"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb","Type":"ContainerDied","Data":"860e58293c6ab4000ace6a7a08af54588451d23791dc8c47332dfb4789008dd1"}
Sep 10 17:38:16.799418 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.799392    2568 generic.go:356] "Generic (PLEG): container finished" podID="79636d0d-de6b-441f-a454-97bb355c6bde" containerID="6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3" exitCode=0
Sep 10 17:38:16.799542 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.799421    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"
Sep 10 17:38:16.799542 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.799476    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7" event={"ID":"79636d0d-de6b-441f-a454-97bb355c6bde","Type":"ContainerDied","Data":"6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3"}
Sep 10 17:38:16.799542 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.799526    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7" event={"ID":"79636d0d-de6b-441f-a454-97bb355c6bde","Type":"ContainerDied","Data":"57f2ca24e90f94c5b640abd6269a4a65bff2ae4ef1af1c1e87f61d683ecc35e6"}
Sep 10 17:38:16.805828 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.805804    2568 scope.go:117] "RemoveContainer" containerID="440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913"
Sep 10 17:38:16.806135 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:16.806116    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913\": container with ID starting with 440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913 not found: ID does not exist" containerID="440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913"
Sep 10 17:38:16.806192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.806143    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913"} err="failed to get container status \"440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913\": rpc error: code = NotFound desc = could not find container \"440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913\": container with ID starting with 440d4a8574e035887821b65f9d653f430c3ddbd73d983042e2e4fb1a8c4e0913 not found: ID does not exist"
Sep 10 17:38:16.806192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.806159    2568 scope.go:117] "RemoveContainer" containerID="e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7"
Sep 10 17:38:16.814002 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.813985    2568 scope.go:117] "RemoveContainer" containerID="e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7"
Sep 10 17:38:16.814259 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:16.814226    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7\": container with ID starting with e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7 not found: ID does not exist" containerID="e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7"
Sep 10 17:38:16.814313 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.814265    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7"} err="failed to get container status \"e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7\": rpc error: code = NotFound desc = could not find container \"e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7\": container with ID starting with e27755b04c2b9a575bac660c48caa661d30c809c19b1ef3396f8b99ea81687b7 not found: ID does not exist"
Sep 10 17:38:16.814313 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.814282    2568 scope.go:117] "RemoveContainer" containerID="6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3"
Sep 10 17:38:16.821665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.821647    2568 scope.go:117] "RemoveContainer" containerID="6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3"
Sep 10 17:38:16.821936 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:16.821919    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3\": container with ID starting with 6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3 not found: ID does not exist" containerID="6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3"
Sep 10 17:38:16.821993 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.821943    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3"} err="failed to get container status \"6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3\": rpc error: code = NotFound desc = could not find container \"6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3\": container with ID starting with 6a4fa52fd530a3f347b3c6fbc63f4bc68e1b11adabd0bed870e682260749a0b3 not found: ID does not exist"
Sep 10 17:38:16.868255 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868206    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr4gj\" (UniqueName: \"kubernetes.io/projected/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-kube-api-access-hr4gj\") pod \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\" (UID: \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\") "
Sep 10 17:38:16.868391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868273    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5pkg\" (UniqueName: \"kubernetes.io/projected/d38f61de-7617-4e38-ab7c-47b7a68f7c26-kube-api-access-t5pkg\") pod \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") "
Sep 10 17:38:16.868391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868373    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38f61de-7617-4e38-ab7c-47b7a68f7c26-openshift-service-ca\") pod \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") "
Sep 10 17:38:16.868463 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868399    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79636d0d-de6b-441f-a454-97bb355c6bde-plugins-config-volume\") pod \"79636d0d-de6b-441f-a454-97bb355c6bde\" (UID: \"79636d0d-de6b-441f-a454-97bb355c6bde\") "
Sep 10 17:38:16.868463 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868444    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-metrics-certs\") pod \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") "
Sep 10 17:38:16.868553 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868478    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7ttl\" (UniqueName: \"kubernetes.io/projected/79636d0d-de6b-441f-a454-97bb355c6bde-kube-api-access-h7ttl\") pod \"79636d0d-de6b-441f-a454-97bb355c6bde\" (UID: \"79636d0d-de6b-441f-a454-97bb355c6bde\") "
Sep 10 17:38:16.868553 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868513    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls\") pod \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\" (UID: \"d38f61de-7617-4e38-ab7c-47b7a68f7c26\") "
Sep 10 17:38:16.868660 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868554    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-plugins-config-volume\") pod \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\" (UID: \"020e9b23-ccf5-4181-a6c9-9e2f22b61cdb\") "
Sep 10 17:38:16.868816 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868770    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38f61de-7617-4e38-ab7c-47b7a68f7c26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d38f61de-7617-4e38-ab7c-47b7a68f7c26" (UID: "d38f61de-7617-4e38-ab7c-47b7a68f7c26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:16.868898 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868845    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79636d0d-de6b-441f-a454-97bb355c6bde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79636d0d-de6b-441f-a454-97bb355c6bde" (UID: "79636d0d-de6b-441f-a454-97bb355c6bde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:16.868958 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.868921    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38f61de-7617-4e38-ab7c-47b7a68f7c26-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:16.869138 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.869110    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "020e9b23-ccf5-4181-a6c9-9e2f22b61cdb" (UID: "020e9b23-ccf5-4181-a6c9-9e2f22b61cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:16.870583 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.870563    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38f61de-7617-4e38-ab7c-47b7a68f7c26-kube-api-access-t5pkg" (OuterVolumeSpecName: "kube-api-access-t5pkg") pod "d38f61de-7617-4e38-ab7c-47b7a68f7c26" (UID: "d38f61de-7617-4e38-ab7c-47b7a68f7c26"). InnerVolumeSpecName "kube-api-access-t5pkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:16.870688 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.870596    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79636d0d-de6b-441f-a454-97bb355c6bde-kube-api-access-h7ttl" (OuterVolumeSpecName: "kube-api-access-h7ttl") pod "79636d0d-de6b-441f-a454-97bb355c6bde" (UID: "79636d0d-de6b-441f-a454-97bb355c6bde"). InnerVolumeSpecName "kube-api-access-h7ttl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:16.870739 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.870699    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d38f61de-7617-4e38-ab7c-47b7a68f7c26" (UID: "d38f61de-7617-4e38-ab7c-47b7a68f7c26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:16.870785 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.870729    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-kube-api-access-hr4gj" (OuterVolumeSpecName: "kube-api-access-hr4gj") pod "020e9b23-ccf5-4181-a6c9-9e2f22b61cdb" (UID: "020e9b23-ccf5-4181-a6c9-9e2f22b61cdb"). InnerVolumeSpecName "kube-api-access-hr4gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:16.870949 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.870930    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d38f61de-7617-4e38-ab7c-47b7a68f7c26" (UID: "d38f61de-7617-4e38-ab7c-47b7a68f7c26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:16.970391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.970312    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ttl\" (UniqueName: \"kubernetes.io/projected/79636d0d-de6b-441f-a454-97bb355c6bde-kube-api-access-h7ttl\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:16.970391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.970338    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:16.970391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.970349    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:16.970391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.970357    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr4gj\" (UniqueName: \"kubernetes.io/projected/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb-kube-api-access-hr4gj\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:16.970391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.970365    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5pkg\" (UniqueName: \"kubernetes.io/projected/d38f61de-7617-4e38-ab7c-47b7a68f7c26-kube-api-access-t5pkg\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:16.970391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.970373    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79636d0d-de6b-441f-a454-97bb355c6bde-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:16.970391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:16.970381    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38f61de-7617-4e38-ab7c-47b7a68f7c26-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:17.122130 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.122105    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"]
Sep 10 17:38:17.127106 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.127083    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0b7bb86-b96c86995-j8gjq"]
Sep 10 17:38:17.139810 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.139788    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"]
Sep 10 17:38:17.146664 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.146640    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0b7bb86-786bd857b6-kmpw7"]
Sep 10 17:38:17.154733 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.154710    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"]
Sep 10 17:38:17.157667 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.157646    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0b7bb86-56b977d665-rtp7k"]
Sep 10 17:38:17.770305 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.770284    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020e9b23-ccf5-4181-a6c9-9e2f22b61cdb" path="/var/lib/kubelet/pods/020e9b23-ccf5-4181-a6c9-9e2f22b61cdb/volumes"
Sep 10 17:38:17.770652 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.770637    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79636d0d-de6b-441f-a454-97bb355c6bde" path="/var/lib/kubelet/pods/79636d0d-de6b-441f-a454-97bb355c6bde/volumes"
Sep 10 17:38:17.771021 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:17.770998    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38f61de-7617-4e38-ab7c-47b7a68f7c26" path="/var/lib/kubelet/pods/d38f61de-7617-4e38-ab7c-47b7a68f7c26/volumes"
Sep 10 17:38:22.556132 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.556096    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"]
Sep 10 17:38:22.556793 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.556768    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020e9b23-ccf5-4181-a6c9-9e2f22b61cdb" containerName="payload-processing"
Sep 10 17:38:22.556949 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.556929    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="020e9b23-ccf5-4181-a6c9-9e2f22b61cdb" containerName="payload-processing"
Sep 10 17:38:22.557081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.556970    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38f61de-7617-4e38-ab7c-47b7a68f7c26" containerName="maas-api"
Sep 10 17:38:22.557081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.556980    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38f61de-7617-4e38-ab7c-47b7a68f7c26" containerName="maas-api"
Sep 10 17:38:22.557081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.556993    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79636d0d-de6b-441f-a454-97bb355c6bde" containerName="payload-pre-processing"
Sep 10 17:38:22.557081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.557003    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="79636d0d-de6b-441f-a454-97bb355c6bde" containerName="payload-pre-processing"
Sep 10 17:38:22.557284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.557104    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38f61de-7617-4e38-ab7c-47b7a68f7c26" containerName="maas-api"
Sep 10 17:38:22.557284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.557115    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="020e9b23-ccf5-4181-a6c9-9e2f22b61cdb" containerName="payload-processing"
Sep 10 17:38:22.557284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.557126    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="79636d0d-de6b-441f-a454-97bb355c6bde" containerName="payload-pre-processing"
Sep 10 17:38:22.560650 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.560634    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.563392 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.563372    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e67dcae3-serving-cert\""
Sep 10 17:38:22.571842 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.570851    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"]
Sep 10 17:38:22.614456 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.614423    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-metrics-certs\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.614627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.614467    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f40439-2f63-4214-b27e-a09b6ccede19-openshift-service-ca\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.614627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.614492    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khw4r\" (UniqueName: \"kubernetes.io/projected/38f40439-2f63-4214-b27e-a09b6ccede19-kube-api-access-khw4r\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.614627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.614559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.715465 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.715424    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.715640 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.715543    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-metrics-certs\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.715640 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.715584    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f40439-2f63-4214-b27e-a09b6ccede19-openshift-service-ca\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.715640 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:22.715595    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-serving-cert: secret "maas-api-e2e-derive-e67dcae3-serving-cert" not found
Sep 10 17:38:22.715640 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.715618    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khw4r\" (UniqueName: \"kubernetes.io/projected/38f40439-2f63-4214-b27e-a09b6ccede19-kube-api-access-khw4r\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.715867 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:22.715671    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls podName:38f40439-2f63-4214-b27e-a09b6ccede19 nodeName:}" failed. No retries permitted until 2026-09-10 17:38:23.215651124 +0000 UTC m=+871.925245387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls") pod "maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5" (UID: "38f40439-2f63-4214-b27e-a09b6ccede19") : secret "maas-api-e2e-derive-e67dcae3-serving-cert" not found
Sep 10 17:38:22.716307 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.716288    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f40439-2f63-4214-b27e-a09b6ccede19-openshift-service-ca\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.718015 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.717995    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-metrics-certs\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:22.730637 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:22.730615    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khw4r\" (UniqueName: \"kubernetes.io/projected/38f40439-2f63-4214-b27e-a09b6ccede19-kube-api-access-khw4r\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:23.219918 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.219878    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:23.222490 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.222457    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls\") pod \"maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:23.224339 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.224318    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"]
Sep 10 17:38:23.227908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.227885    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:23.230525 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.230503    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e67dcae3\""
Sep 10 17:38:23.230739 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.230713    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"]
Sep 10 17:38:23.234263 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.234245    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:23.236723 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.236710    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e67dcae3-dockercfg-7lq79\""
Sep 10 17:38:23.243913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.243845    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"]
Sep 10 17:38:23.248821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.248770    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"]
Sep 10 17:38:23.320310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.320272    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3101d49-6b71-4f5f-b57b-73256cb38b6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr\" (UID: \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:23.320484 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.320329    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4487n\" (UniqueName: \"kubernetes.io/projected/f3101d49-6b71-4f5f-b57b-73256cb38b6a-kube-api-access-4487n\") pod \"payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr\" (UID: \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:23.320484 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.320412    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzm2x\" (UniqueName: \"kubernetes.io/projected/474a4767-b1cc-498c-833b-2f06b5543957-kube-api-access-pzm2x\") pod \"payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68\" (UID: \"474a4767-b1cc-498c-833b-2f06b5543957\") " pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:23.320484 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.320445    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474a4767-b1cc-498c-833b-2f06b5543957-plugins-config-volume\") pod \"payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68\" (UID: \"474a4767-b1cc-498c-833b-2f06b5543957\") " pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:23.421510 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.421474    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4487n\" (UniqueName: \"kubernetes.io/projected/f3101d49-6b71-4f5f-b57b-73256cb38b6a-kube-api-access-4487n\") pod \"payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr\" (UID: \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:23.421680 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.421531    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzm2x\" (UniqueName: \"kubernetes.io/projected/474a4767-b1cc-498c-833b-2f06b5543957-kube-api-access-pzm2x\") pod \"payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68\" (UID: \"474a4767-b1cc-498c-833b-2f06b5543957\") " pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:23.421680 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.421572    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474a4767-b1cc-498c-833b-2f06b5543957-plugins-config-volume\") pod \"payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68\" (UID: \"474a4767-b1cc-498c-833b-2f06b5543957\") " pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:23.421680 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.421611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3101d49-6b71-4f5f-b57b-73256cb38b6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr\" (UID: \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:23.422414 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.422396    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474a4767-b1cc-498c-833b-2f06b5543957-plugins-config-volume\") pod \"payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68\" (UID: \"474a4767-b1cc-498c-833b-2f06b5543957\") " pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:23.422452 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.422403    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3101d49-6b71-4f5f-b57b-73256cb38b6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr\" (UID: \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:23.430382 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.430351    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4487n\" (UniqueName: \"kubernetes.io/projected/f3101d49-6b71-4f5f-b57b-73256cb38b6a-kube-api-access-4487n\") pod \"payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr\" (UID: \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:23.440000 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.439970    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzm2x\" (UniqueName: \"kubernetes.io/projected/474a4767-b1cc-498c-833b-2f06b5543957-kube-api-access-pzm2x\") pod \"payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68\" (UID: \"474a4767-b1cc-498c-833b-2f06b5543957\") " pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:23.475611 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.475518    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:23.545116 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.544805    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:23.552830 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.552801    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:23.747282 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.747116    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"]
Sep 10 17:38:23.750267 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:38:23.750218    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f40439_2f63_4214_b27e_a09b6ccede19.slice/crio-1645ef21253efc5ef40cd6f8c9899d4f2866c8c8c8da3b27f39b112c45b8e060 WatchSource:0}: Error finding container 1645ef21253efc5ef40cd6f8c9899d4f2866c8c8c8da3b27f39b112c45b8e060: Status 404 returned error can't find the container with id 1645ef21253efc5ef40cd6f8c9899d4f2866c8c8c8da3b27f39b112c45b8e060
Sep 10 17:38:23.824943 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.824919    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"]
Sep 10 17:38:23.826993 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:38:23.826968    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3101d49_6b71_4f5f_b57b_73256cb38b6a.slice/crio-2ebd2255acbb55b0b06abef174674523dac686a801618553793f893a1aaed77e WatchSource:0}: Error finding container 2ebd2255acbb55b0b06abef174674523dac686a801618553793f893a1aaed77e: Status 404 returned error can't find the container with id 2ebd2255acbb55b0b06abef174674523dac686a801618553793f893a1aaed77e
Sep 10 17:38:23.839792 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.839759    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr" event={"ID":"f3101d49-6b71-4f5f-b57b-73256cb38b6a","Type":"ContainerStarted","Data":"2ebd2255acbb55b0b06abef174674523dac686a801618553793f893a1aaed77e"}
Sep 10 17:38:23.840972 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.840940    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5" event={"ID":"38f40439-2f63-4214-b27e-a09b6ccede19","Type":"ContainerStarted","Data":"1645ef21253efc5ef40cd6f8c9899d4f2866c8c8c8da3b27f39b112c45b8e060"}
Sep 10 17:38:23.855132 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:23.855100    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"]
Sep 10 17:38:23.856800 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:38:23.856771    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474a4767_b1cc_498c_833b_2f06b5543957.slice/crio-3addaec8992526c6b09ccda9f426bc72af4472c733c3eb47ad6e790d77090d56 WatchSource:0}: Error finding container 3addaec8992526c6b09ccda9f426bc72af4472c733c3eb47ad6e790d77090d56: Status 404 returned error can't find the container with id 3addaec8992526c6b09ccda9f426bc72af4472c733c3eb47ad6e790d77090d56
Sep 10 17:38:24.849355 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.849312    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr" event={"ID":"f3101d49-6b71-4f5f-b57b-73256cb38b6a","Type":"ContainerStarted","Data":"5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb"}
Sep 10 17:38:24.849823 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.849431    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:24.851264 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.851216    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5" event={"ID":"38f40439-2f63-4214-b27e-a09b6ccede19","Type":"ContainerStarted","Data":"9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3"}
Sep 10 17:38:24.851389 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.851283    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:24.852869 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.852846    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68" event={"ID":"474a4767-b1cc-498c-833b-2f06b5543957","Type":"ContainerStarted","Data":"776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30"}
Sep 10 17:38:24.852984 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.852883    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68" event={"ID":"474a4767-b1cc-498c-833b-2f06b5543957","Type":"ContainerStarted","Data":"3addaec8992526c6b09ccda9f426bc72af4472c733c3eb47ad6e790d77090d56"}
Sep 10 17:38:24.853048 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.853026    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:24.868267 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.868192    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr" podStartSLOduration=1.868176755 podStartE2EDuration="1.868176755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:24.86813005 +0000 UTC m=+873.577724317" watchObservedRunningTime="2026-09-10 17:38:24.868176755 +0000 UTC m=+873.577771010"
Sep 10 17:38:24.889622 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:24.889567    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5" podStartSLOduration=2.5293145949999998 podStartE2EDuration="2.889550473s" totalImagesPullingTime="360.235878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:24.886906578 +0000 UTC m=+873.596500846" watchObservedRunningTime="2026-09-10 17:38:24.889550473 +0000 UTC m=+873.599144739"
Sep 10 17:38:30.863021 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:30.862991    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:30.883821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:30.883778    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68" podStartSLOduration=7.88376371 podStartE2EDuration="7.88376371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:24.904369497 +0000 UTC m=+873.613963763" watchObservedRunningTime="2026-09-10 17:38:30.88376371 +0000 UTC m=+879.593357975"
Sep 10 17:38:35.859982 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:35.859946    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:35.860439 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:35.860418    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:44.555364 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.555285    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"]
Sep 10 17:38:44.555812 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.555540    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts" podUID="d6892ca0-8bb8-48c7-9b9d-952632b5e42c" containerName="maas-api" containerID="cri-o://04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312" gracePeriod=30
Sep 10 17:38:44.623506 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.623468    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"]
Sep 10 17:38:44.623749 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.623715    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8" podUID="1963dba5-19a1-4c21-b1a2-5af18c06261a" containerName="payload-pre-processing" containerID="cri-o://c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7" gracePeriod=30
Sep 10 17:38:44.873405 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.873378    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:38:44.892784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.892761    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:38:44.960588 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.960551    2568 generic.go:356] "Generic (PLEG): container finished" podID="1963dba5-19a1-4c21-b1a2-5af18c06261a" containerID="c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7" exitCode=0
Sep 10 17:38:44.960775 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.960625    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"
Sep 10 17:38:44.960775 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.960628    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8" event={"ID":"1963dba5-19a1-4c21-b1a2-5af18c06261a","Type":"ContainerDied","Data":"c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7"}
Sep 10 17:38:44.960775 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.960719    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8" event={"ID":"1963dba5-19a1-4c21-b1a2-5af18c06261a","Type":"ContainerDied","Data":"8e77f2625c4e946e20b889c76596204eab875bb10abd9b441173442d21a50e5e"}
Sep 10 17:38:44.960775 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.960749    2568 scope.go:117] "RemoveContainer" containerID="c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7"
Sep 10 17:38:44.962053 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.962034    2568 generic.go:356] "Generic (PLEG): container finished" podID="d6892ca0-8bb8-48c7-9b9d-952632b5e42c" containerID="04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312" exitCode=0
Sep 10 17:38:44.962124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.962076    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts" event={"ID":"d6892ca0-8bb8-48c7-9b9d-952632b5e42c","Type":"ContainerDied","Data":"04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312"}
Sep 10 17:38:44.962124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.962084    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"
Sep 10 17:38:44.962124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.962107    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts" event={"ID":"d6892ca0-8bb8-48c7-9b9d-952632b5e42c","Type":"ContainerDied","Data":"b71ffb53a7635651b4736622f00bbd49e40c83d794ccc751f7ec44dc024ffbae"}
Sep 10 17:38:44.969653 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.969630    2568 scope.go:117] "RemoveContainer" containerID="c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7"
Sep 10 17:38:44.969925 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:44.969903    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7\": container with ID starting with c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7 not found: ID does not exist" containerID="c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7"
Sep 10 17:38:44.969991 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.969932    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7"} err="failed to get container status \"c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7\": rpc error: code = NotFound desc = could not find container \"c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7\": container with ID starting with c2c693a18c293e0f0bce9362709c6135f850ff16d86c8e3538142cf7e9680fe7 not found: ID does not exist"
Sep 10 17:38:44.969991 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.969949    2568 scope.go:117] "RemoveContainer" containerID="04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312"
Sep 10 17:38:44.978164 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.978142    2568 scope.go:117] "RemoveContainer" containerID="04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312"
Sep 10 17:38:44.978462 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:44.978443    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312\": container with ID starting with 04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312 not found: ID does not exist" containerID="04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312"
Sep 10 17:38:44.978541 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:44.978472    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312"} err="failed to get container status \"04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312\": rpc error: code = NotFound desc = could not find container \"04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312\": container with ID starting with 04dffcf9192c0222effe385f51c6550a2c94b53a6ea5d34c9f206d7a789ed312 not found: ID does not exist"
Sep 10 17:38:45.024806 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.024777    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-openshift-service-ca\") pod \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") "
Sep 10 17:38:45.024921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.024831    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1963dba5-19a1-4c21-b1a2-5af18c06261a-plugins-config-volume\") pod \"1963dba5-19a1-4c21-b1a2-5af18c06261a\" (UID: \"1963dba5-19a1-4c21-b1a2-5af18c06261a\") "
Sep 10 17:38:45.024921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.024854    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqpdm\" (UniqueName: \"kubernetes.io/projected/1963dba5-19a1-4c21-b1a2-5af18c06261a-kube-api-access-wqpdm\") pod \"1963dba5-19a1-4c21-b1a2-5af18c06261a\" (UID: \"1963dba5-19a1-4c21-b1a2-5af18c06261a\") "
Sep 10 17:38:45.025037 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.024957    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-metrics-certs\") pod \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") "
Sep 10 17:38:45.025037 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.025017    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls\") pod \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") "
Sep 10 17:38:45.025153 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.025056    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqr7d\" (UniqueName: \"kubernetes.io/projected/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-kube-api-access-bqr7d\") pod \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\" (UID: \"d6892ca0-8bb8-48c7-9b9d-952632b5e42c\") "
Sep 10 17:38:45.025205 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.025174    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6892ca0-8bb8-48c7-9b9d-952632b5e42c" (UID: "d6892ca0-8bb8-48c7-9b9d-952632b5e42c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:45.025291 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.025217    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1963dba5-19a1-4c21-b1a2-5af18c06261a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1963dba5-19a1-4c21-b1a2-5af18c06261a" (UID: "1963dba5-19a1-4c21-b1a2-5af18c06261a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:45.025417 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.025399    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:45.025467 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.025422    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1963dba5-19a1-4c21-b1a2-5af18c06261a-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:45.027104 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.027078    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6892ca0-8bb8-48c7-9b9d-952632b5e42c" (UID: "d6892ca0-8bb8-48c7-9b9d-952632b5e42c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:45.027243 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.027211    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-kube-api-access-bqr7d" (OuterVolumeSpecName: "kube-api-access-bqr7d") pod "d6892ca0-8bb8-48c7-9b9d-952632b5e42c" (UID: "d6892ca0-8bb8-48c7-9b9d-952632b5e42c"). InnerVolumeSpecName "kube-api-access-bqr7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:45.027445 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.027426    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6892ca0-8bb8-48c7-9b9d-952632b5e42c" (UID: "d6892ca0-8bb8-48c7-9b9d-952632b5e42c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:45.027502 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.027428    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1963dba5-19a1-4c21-b1a2-5af18c06261a-kube-api-access-wqpdm" (OuterVolumeSpecName: "kube-api-access-wqpdm") pod "1963dba5-19a1-4c21-b1a2-5af18c06261a" (UID: "1963dba5-19a1-4c21-b1a2-5af18c06261a"). InnerVolumeSpecName "kube-api-access-wqpdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:45.126477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.126400    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:45.126477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.126429    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqr7d\" (UniqueName: \"kubernetes.io/projected/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-kube-api-access-bqr7d\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:45.126477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.126440    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqpdm\" (UniqueName: \"kubernetes.io/projected/1963dba5-19a1-4c21-b1a2-5af18c06261a-kube-api-access-wqpdm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:45.126477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.126448    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6892ca0-8bb8-48c7-9b9d-952632b5e42c-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:45.289901 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.289874    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"]
Sep 10 17:38:45.293551 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.293527    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-794622-b7cbcf4cd-p96t8"]
Sep 10 17:38:45.305048 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.305017    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"]
Sep 10 17:38:45.310092 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.310071    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-794622-c8964fd77-cfrts"]
Sep 10 17:38:45.765921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.765881    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1963dba5-19a1-4c21-b1a2-5af18c06261a" path="/var/lib/kubelet/pods/1963dba5-19a1-4c21-b1a2-5af18c06261a/volumes"
Sep 10 17:38:45.766495 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:45.766476    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6892ca0-8bb8-48c7-9b9d-952632b5e42c" path="/var/lib/kubelet/pods/d6892ca0-8bb8-48c7-9b9d-952632b5e42c/volumes"
Sep 10 17:38:51.752514 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:51.752478    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:38:51.754762 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:51.754734    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:38:53.236854 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.236823    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"]
Sep 10 17:38:53.237367 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.237023    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5" podUID="38f40439-2f63-4214-b27e-a09b6ccede19" containerName="maas-api" containerID="cri-o://9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3" gracePeriod=30
Sep 10 17:38:53.296726 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.296693    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"]
Sep 10 17:38:53.296991 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.296936    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68" podUID="474a4767-b1cc-498c-833b-2f06b5543957" containerName="payload-processing" containerID="cri-o://776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30" gracePeriod=30
Sep 10 17:38:53.315878 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.315846    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"]
Sep 10 17:38:53.316068 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.316043    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr" podUID="f3101d49-6b71-4f5f-b57b-73256cb38b6a" containerName="payload-pre-processing" containerID="cri-o://5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb" gracePeriod=30
Sep 10 17:38:53.586053 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.586027    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:53.614312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.614038    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:53.625789 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.625745    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:53.708387 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.708354    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f40439-2f63-4214-b27e-a09b6ccede19-openshift-service-ca\") pod \"38f40439-2f63-4214-b27e-a09b6ccede19\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") "
Sep 10 17:38:53.708387 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.708387    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khw4r\" (UniqueName: \"kubernetes.io/projected/38f40439-2f63-4214-b27e-a09b6ccede19-kube-api-access-khw4r\") pod \"38f40439-2f63-4214-b27e-a09b6ccede19\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") "
Sep 10 17:38:53.708626 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.708437    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-metrics-certs\") pod \"38f40439-2f63-4214-b27e-a09b6ccede19\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") "
Sep 10 17:38:53.708626 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.708457    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3101d49-6b71-4f5f-b57b-73256cb38b6a-plugins-config-volume\") pod \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\" (UID: \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\") "
Sep 10 17:38:53.708626 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.708478    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4487n\" (UniqueName: \"kubernetes.io/projected/f3101d49-6b71-4f5f-b57b-73256cb38b6a-kube-api-access-4487n\") pod \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\" (UID: \"f3101d49-6b71-4f5f-b57b-73256cb38b6a\") "
Sep 10 17:38:53.708626 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.708500    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls\") pod \"38f40439-2f63-4214-b27e-a09b6ccede19\" (UID: \"38f40439-2f63-4214-b27e-a09b6ccede19\") "
Sep 10 17:38:53.708860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.708773    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f40439-2f63-4214-b27e-a09b6ccede19-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38f40439-2f63-4214-b27e-a09b6ccede19" (UID: "38f40439-2f63-4214-b27e-a09b6ccede19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:53.708921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.708872    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3101d49-6b71-4f5f-b57b-73256cb38b6a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3101d49-6b71-4f5f-b57b-73256cb38b6a" (UID: "f3101d49-6b71-4f5f-b57b-73256cb38b6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:53.710661 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.710626    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3101d49-6b71-4f5f-b57b-73256cb38b6a-kube-api-access-4487n" (OuterVolumeSpecName: "kube-api-access-4487n") pod "f3101d49-6b71-4f5f-b57b-73256cb38b6a" (UID: "f3101d49-6b71-4f5f-b57b-73256cb38b6a"). InnerVolumeSpecName "kube-api-access-4487n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:53.710661 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.710630    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38f40439-2f63-4214-b27e-a09b6ccede19" (UID: "38f40439-2f63-4214-b27e-a09b6ccede19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:53.710832 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.710678    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f40439-2f63-4214-b27e-a09b6ccede19-kube-api-access-khw4r" (OuterVolumeSpecName: "kube-api-access-khw4r") pod "38f40439-2f63-4214-b27e-a09b6ccede19" (UID: "38f40439-2f63-4214-b27e-a09b6ccede19"). InnerVolumeSpecName "kube-api-access-khw4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:53.710832 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.710686    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "38f40439-2f63-4214-b27e-a09b6ccede19" (UID: "38f40439-2f63-4214-b27e-a09b6ccede19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:53.809718 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.809654    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzm2x\" (UniqueName: \"kubernetes.io/projected/474a4767-b1cc-498c-833b-2f06b5543957-kube-api-access-pzm2x\") pod \"474a4767-b1cc-498c-833b-2f06b5543957\" (UID: \"474a4767-b1cc-498c-833b-2f06b5543957\") "
Sep 10 17:38:53.809718 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.809693    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474a4767-b1cc-498c-833b-2f06b5543957-plugins-config-volume\") pod \"474a4767-b1cc-498c-833b-2f06b5543957\" (UID: \"474a4767-b1cc-498c-833b-2f06b5543957\") "
Sep 10 17:38:53.809891 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.809878    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f40439-2f63-4214-b27e-a09b6ccede19-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:53.809935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.809894    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khw4r\" (UniqueName: \"kubernetes.io/projected/38f40439-2f63-4214-b27e-a09b6ccede19-kube-api-access-khw4r\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:53.809935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.809903    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:53.809935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.809911    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3101d49-6b71-4f5f-b57b-73256cb38b6a-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:53.809935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.809920    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4487n\" (UniqueName: \"kubernetes.io/projected/f3101d49-6b71-4f5f-b57b-73256cb38b6a-kube-api-access-4487n\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:53.809935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.809928    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f40439-2f63-4214-b27e-a09b6ccede19-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:53.810094 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.810038    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474a4767-b1cc-498c-833b-2f06b5543957-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "474a4767-b1cc-498c-833b-2f06b5543957" (UID: "474a4767-b1cc-498c-833b-2f06b5543957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:38:53.811608 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.811588    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474a4767-b1cc-498c-833b-2f06b5543957-kube-api-access-pzm2x" (OuterVolumeSpecName: "kube-api-access-pzm2x") pod "474a4767-b1cc-498c-833b-2f06b5543957" (UID: "474a4767-b1cc-498c-833b-2f06b5543957"). InnerVolumeSpecName "kube-api-access-pzm2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:53.910574 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.910536    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzm2x\" (UniqueName: \"kubernetes.io/projected/474a4767-b1cc-498c-833b-2f06b5543957-kube-api-access-pzm2x\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:53.910574 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:53.910567    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474a4767-b1cc-498c-833b-2f06b5543957-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:38:54.009952 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.009919    2568 generic.go:356] "Generic (PLEG): container finished" podID="f3101d49-6b71-4f5f-b57b-73256cb38b6a" containerID="5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb" exitCode=0
Sep 10 17:38:54.010166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.009971    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"
Sep 10 17:38:54.010166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.010014    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr" event={"ID":"f3101d49-6b71-4f5f-b57b-73256cb38b6a","Type":"ContainerDied","Data":"5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb"}
Sep 10 17:38:54.010166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.010075    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr" event={"ID":"f3101d49-6b71-4f5f-b57b-73256cb38b6a","Type":"ContainerDied","Data":"2ebd2255acbb55b0b06abef174674523dac686a801618553793f893a1aaed77e"}
Sep 10 17:38:54.010166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.010101    2568 scope.go:117] "RemoveContainer" containerID="5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb"
Sep 10 17:38:54.011541 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.011509    2568 generic.go:356] "Generic (PLEG): container finished" podID="38f40439-2f63-4214-b27e-a09b6ccede19" containerID="9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3" exitCode=0
Sep 10 17:38:54.011719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.011560    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"
Sep 10 17:38:54.011719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.011612    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5" event={"ID":"38f40439-2f63-4214-b27e-a09b6ccede19","Type":"ContainerDied","Data":"9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3"}
Sep 10 17:38:54.011719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.011674    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5" event={"ID":"38f40439-2f63-4214-b27e-a09b6ccede19","Type":"ContainerDied","Data":"1645ef21253efc5ef40cd6f8c9899d4f2866c8c8c8da3b27f39b112c45b8e060"}
Sep 10 17:38:54.013302 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.013276    2568 generic.go:356] "Generic (PLEG): container finished" podID="474a4767-b1cc-498c-833b-2f06b5543957" containerID="776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30" exitCode=0
Sep 10 17:38:54.013397 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.013310    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"
Sep 10 17:38:54.013397 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.013338    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68" event={"ID":"474a4767-b1cc-498c-833b-2f06b5543957","Type":"ContainerDied","Data":"776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30"}
Sep 10 17:38:54.013397 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.013377    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68" event={"ID":"474a4767-b1cc-498c-833b-2f06b5543957","Type":"ContainerDied","Data":"3addaec8992526c6b09ccda9f426bc72af4472c733c3eb47ad6e790d77090d56"}
Sep 10 17:38:54.020956 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.020932    2568 scope.go:117] "RemoveContainer" containerID="5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb"
Sep 10 17:38:54.021255 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:54.021216    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb\": container with ID starting with 5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb not found: ID does not exist" containerID="5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb"
Sep 10 17:38:54.021343 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.021265    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb"} err="failed to get container status \"5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb\": rpc error: code = NotFound desc = could not find container \"5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb\": container with ID starting with 5b251a119cd4a1eeb4b0b597b31b19547ed098b8a4984a9638343d2e2c03c6cb not found: ID does not exist"
Sep 10 17:38:54.021343 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.021289    2568 scope.go:117] "RemoveContainer" containerID="9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3"
Sep 10 17:38:54.029961 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.029855    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"]
Sep 10 17:38:54.030014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.029984    2568 scope.go:117] "RemoveContainer" containerID="9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3"
Sep 10 17:38:54.030378 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:54.030358    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3\": container with ID starting with 9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3 not found: ID does not exist" containerID="9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3"
Sep 10 17:38:54.030495 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.030381    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3"} err="failed to get container status \"9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3\": rpc error: code = NotFound desc = could not find container \"9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3\": container with ID starting with 9cd26afcbac56ba5a6aceecf445e299aadf5f8f13fdd8b840cac94e9460e6de3 not found: ID does not exist"
Sep 10 17:38:54.030495 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.030398    2568 scope.go:117] "RemoveContainer" containerID="776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30"
Sep 10 17:38:54.034943 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.034919    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e67dcae3-77bbbb665c-hqmv5"]
Sep 10 17:38:54.039096 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.039078    2568 scope.go:117] "RemoveContainer" containerID="776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30"
Sep 10 17:38:54.039409 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:54.039389    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30\": container with ID starting with 776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30 not found: ID does not exist" containerID="776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30"
Sep 10 17:38:54.039483 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.039415    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30"} err="failed to get container status \"776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30\": rpc error: code = NotFound desc = could not find container \"776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30\": container with ID starting with 776de7781acfa484f92e9243398e54c754c28949c84ac91b762cbefcd5014a30 not found: ID does not exist"
Sep 10 17:38:54.053376 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.053350    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"]
Sep 10 17:38:54.056989 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:54.056957    2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474a4767_b1cc_498c_833b_2f06b5543957.slice/crio-3addaec8992526c6b09ccda9f426bc72af4472c733c3eb47ad6e790d77090d56\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474a4767_b1cc_498c_833b_2f06b5543957.slice\": RecentStats: unable to find data in memory cache]"
Sep 10 17:38:54.057707 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.057625    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e67dcae3-856d94b8cd-gcpbr"]
Sep 10 17:38:54.067452 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.067400    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"]
Sep 10 17:38:54.072831 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:54.072813    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e67dcae3-6d8cf57497-qgb68"]
Sep 10 17:38:55.763907 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:55.763872    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f40439-2f63-4214-b27e-a09b6ccede19" path="/var/lib/kubelet/pods/38f40439-2f63-4214-b27e-a09b6ccede19/volumes"
Sep 10 17:38:55.764302 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:55.764257    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="474a4767-b1cc-498c-833b-2f06b5543957" path="/var/lib/kubelet/pods/474a4767-b1cc-498c-833b-2f06b5543957/volumes"
Sep 10 17:38:55.764580 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:55.764567    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3101d49-6b71-4f5f-b57b-73256cb38b6a" path="/var/lib/kubelet/pods/f3101d49-6b71-4f5f-b57b-73256cb38b6a/volumes"
Sep 10 17:38:56.602547 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.602517    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"]
Sep 10 17:38:56.603185 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603157    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3101d49-6b71-4f5f-b57b-73256cb38b6a" containerName="payload-pre-processing"
Sep 10 17:38:56.603185 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603185    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3101d49-6b71-4f5f-b57b-73256cb38b6a" containerName="payload-pre-processing"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603205    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f40439-2f63-4214-b27e-a09b6ccede19" containerName="maas-api"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603210    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f40439-2f63-4214-b27e-a09b6ccede19" containerName="maas-api"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603245    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1963dba5-19a1-4c21-b1a2-5af18c06261a" containerName="payload-pre-processing"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603255    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1963dba5-19a1-4c21-b1a2-5af18c06261a" containerName="payload-pre-processing"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603270    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474a4767-b1cc-498c-833b-2f06b5543957" containerName="payload-processing"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603278    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="474a4767-b1cc-498c-833b-2f06b5543957" containerName="payload-processing"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603290    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6892ca0-8bb8-48c7-9b9d-952632b5e42c" containerName="maas-api"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603299    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6892ca0-8bb8-48c7-9b9d-952632b5e42c" containerName="maas-api"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603404    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="474a4767-b1cc-498c-833b-2f06b5543957" containerName="payload-processing"
Sep 10 17:38:56.603416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603419    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1963dba5-19a1-4c21-b1a2-5af18c06261a" containerName="payload-pre-processing"
Sep 10 17:38:56.603819 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603428    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f40439-2f63-4214-b27e-a09b6ccede19" containerName="maas-api"
Sep 10 17:38:56.603819 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603441    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6892ca0-8bb8-48c7-9b9d-952632b5e42c" containerName="maas-api"
Sep 10 17:38:56.603819 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.603450    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3101d49-6b71-4f5f-b57b-73256cb38b6a" containerName="payload-pre-processing"
Sep 10 17:38:56.608322 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.608296    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.611123 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.611104    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-14b3e7-serving-cert\""
Sep 10 17:38:56.616461 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.616436    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"]
Sep 10 17:38:56.735102 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.735067    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-metrics-certs\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.735303 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.735147    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.735303 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.735184    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb57d469-efaa-43f2-974f-d25e023d3eae-openshift-service-ca\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.735303 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.735201    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswp5\" (UniqueName: \"kubernetes.io/projected/eb57d469-efaa-43f2-974f-d25e023d3eae-kube-api-access-jswp5\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.836633 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.836594    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.837092 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.836650    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb57d469-efaa-43f2-974f-d25e023d3eae-openshift-service-ca\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.837092 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.836676    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswp5\" (UniqueName: \"kubernetes.io/projected/eb57d469-efaa-43f2-974f-d25e023d3eae-kube-api-access-jswp5\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.837092 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.836711    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-metrics-certs\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.837092 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:56.836754    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-serving-cert: secret "maas-api-e2e-explicit-14b3e7-serving-cert" not found
Sep 10 17:38:56.837092 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:38:56.836819    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls podName:eb57d469-efaa-43f2-974f-d25e023d3eae nodeName:}" failed. No retries permitted until 2026-09-10 17:38:57.336798749 +0000 UTC m=+906.046393011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls") pod "maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq" (UID: "eb57d469-efaa-43f2-974f-d25e023d3eae") : secret "maas-api-e2e-explicit-14b3e7-serving-cert" not found
Sep 10 17:38:56.837482 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.837461    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb57d469-efaa-43f2-974f-d25e023d3eae-openshift-service-ca\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.839179 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.839156    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-metrics-certs\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:56.846813 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:56.846787    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswp5\" (UniqueName: \"kubernetes.io/projected/eb57d469-efaa-43f2-974f-d25e023d3eae-kube-api-access-jswp5\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:57.235845 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.235807    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"]
Sep 10 17:38:57.239630 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.239613    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:57.242634 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.242603    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-14b3e7\""
Sep 10 17:38:57.242787 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.242755    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-14b3e7-dockercfg-2jsfr\""
Sep 10 17:38:57.250114 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.250094    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"]
Sep 10 17:38:57.341014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.340962    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:57.341191 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.341107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cce627-9206-4775-a587-e3d8c44529b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx\" (UID: \"00cce627-9206-4775-a587-e3d8c44529b7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:57.341450 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.341416    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nlx\" (UniqueName: \"kubernetes.io/projected/00cce627-9206-4775-a587-e3d8c44529b7-kube-api-access-s9nlx\") pod \"payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx\" (UID: \"00cce627-9206-4775-a587-e3d8c44529b7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:57.343736 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.343708    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls\") pod \"maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:57.442775 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.442731    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nlx\" (UniqueName: \"kubernetes.io/projected/00cce627-9206-4775-a587-e3d8c44529b7-kube-api-access-s9nlx\") pod \"payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx\" (UID: \"00cce627-9206-4775-a587-e3d8c44529b7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:57.442953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.442920    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cce627-9206-4775-a587-e3d8c44529b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx\" (UID: \"00cce627-9206-4775-a587-e3d8c44529b7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:57.443755 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.443728    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cce627-9206-4775-a587-e3d8c44529b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx\" (UID: \"00cce627-9206-4775-a587-e3d8c44529b7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:57.451793 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.451767    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nlx\" (UniqueName: \"kubernetes.io/projected/00cce627-9206-4775-a587-e3d8c44529b7-kube-api-access-s9nlx\") pod \"payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx\" (UID: \"00cce627-9206-4775-a587-e3d8c44529b7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:57.519650 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.519581    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:57.554973 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.554434    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:57.800410 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.800384    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"]
Sep 10 17:38:57.802227 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:38:57.802202    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb57d469_efaa_43f2_974f_d25e023d3eae.slice/crio-a7a0dda0a7e20122b8ba1bd30643d04cdfe62314e04b3ac4ded1f732a34919fc WatchSource:0}: Error finding container a7a0dda0a7e20122b8ba1bd30643d04cdfe62314e04b3ac4ded1f732a34919fc: Status 404 returned error can't find the container with id a7a0dda0a7e20122b8ba1bd30643d04cdfe62314e04b3ac4ded1f732a34919fc
Sep 10 17:38:57.833283 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:57.833261    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"]
Sep 10 17:38:57.835109 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:38:57.835085    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cce627_9206_4775_a587_e3d8c44529b7.slice/crio-a1674ead86df72bc41d7bafa24c009b6d1302760a5aedcdd4d408e02c923c7fe WatchSource:0}: Error finding container a1674ead86df72bc41d7bafa24c009b6d1302760a5aedcdd4d408e02c923c7fe: Status 404 returned error can't find the container with id a1674ead86df72bc41d7bafa24c009b6d1302760a5aedcdd4d408e02c923c7fe
Sep 10 17:38:58.037593 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:58.037557    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx" event={"ID":"00cce627-9206-4775-a587-e3d8c44529b7","Type":"ContainerStarted","Data":"7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916"}
Sep 10 17:38:58.038031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:58.037617    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx" event={"ID":"00cce627-9206-4775-a587-e3d8c44529b7","Type":"ContainerStarted","Data":"a1674ead86df72bc41d7bafa24c009b6d1302760a5aedcdd4d408e02c923c7fe"}
Sep 10 17:38:58.038031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:58.037760    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:38:58.038968 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:58.038941    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq" event={"ID":"eb57d469-efaa-43f2-974f-d25e023d3eae","Type":"ContainerStarted","Data":"a7a0dda0a7e20122b8ba1bd30643d04cdfe62314e04b3ac4ded1f732a34919fc"}
Sep 10 17:38:58.055594 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:58.055505    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx" podStartSLOduration=1.055488115 podStartE2EDuration="1.055488115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:58.053879755 +0000 UTC m=+906.763474021" watchObservedRunningTime="2026-09-10 17:38:58.055488115 +0000 UTC m=+906.765082384"
Sep 10 17:38:59.047574 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:59.047533    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq" event={"ID":"eb57d469-efaa-43f2-974f-d25e023d3eae","Type":"ContainerStarted","Data":"6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b"}
Sep 10 17:38:59.048026 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:59.047916    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:38:59.065664 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:38:59.065611    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq" podStartSLOduration=2.586717541 podStartE2EDuration="3.065596157s" totalImagesPullingTime="478.878616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:59.064368449 +0000 UTC m=+907.773962717" watchObservedRunningTime="2026-09-10 17:38:59.065596157 +0000 UTC m=+907.775190422"
Sep 10 17:39:05.058218 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:39:05.058184    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:39:09.049622 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:39:09.049591    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:40:02.730124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:02.730091    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"]
Sep 10 17:40:02.730614 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:02.730296    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq" podUID="eb57d469-efaa-43f2-974f-d25e023d3eae" containerName="maas-api" containerID="cri-o://6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b" gracePeriod=30
Sep 10 17:40:02.784124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:02.784092    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"]
Sep 10 17:40:02.784347 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:02.784319    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx" podUID="00cce627-9206-4775-a587-e3d8c44529b7" containerName="payload-pre-processing" containerID="cri-o://7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916" gracePeriod=30
Sep 10 17:40:03.026585 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.026559    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:40:03.059332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.059303    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:40:03.126154 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.126123    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9nlx\" (UniqueName: \"kubernetes.io/projected/00cce627-9206-4775-a587-e3d8c44529b7-kube-api-access-s9nlx\") pod \"00cce627-9206-4775-a587-e3d8c44529b7\" (UID: \"00cce627-9206-4775-a587-e3d8c44529b7\") "
Sep 10 17:40:03.126332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.126174    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jswp5\" (UniqueName: \"kubernetes.io/projected/eb57d469-efaa-43f2-974f-d25e023d3eae-kube-api-access-jswp5\") pod \"eb57d469-efaa-43f2-974f-d25e023d3eae\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") "
Sep 10 17:40:03.126332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.126190    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cce627-9206-4775-a587-e3d8c44529b7-plugins-config-volume\") pod \"00cce627-9206-4775-a587-e3d8c44529b7\" (UID: \"00cce627-9206-4775-a587-e3d8c44529b7\") "
Sep 10 17:40:03.126332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.126213    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls\") pod \"eb57d469-efaa-43f2-974f-d25e023d3eae\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") "
Sep 10 17:40:03.126332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.126279    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-metrics-certs\") pod \"eb57d469-efaa-43f2-974f-d25e023d3eae\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") "
Sep 10 17:40:03.126332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.126306    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb57d469-efaa-43f2-974f-d25e023d3eae-openshift-service-ca\") pod \"eb57d469-efaa-43f2-974f-d25e023d3eae\" (UID: \"eb57d469-efaa-43f2-974f-d25e023d3eae\") "
Sep 10 17:40:03.126646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.126589    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00cce627-9206-4775-a587-e3d8c44529b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00cce627-9206-4775-a587-e3d8c44529b7" (UID: "00cce627-9206-4775-a587-e3d8c44529b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:03.126821 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.126788    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb57d469-efaa-43f2-974f-d25e023d3eae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb57d469-efaa-43f2-974f-d25e023d3eae" (UID: "eb57d469-efaa-43f2-974f-d25e023d3eae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:03.128853 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.128829    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eb57d469-efaa-43f2-974f-d25e023d3eae" (UID: "eb57d469-efaa-43f2-974f-d25e023d3eae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:40:03.128853 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.128841    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cce627-9206-4775-a587-e3d8c44529b7-kube-api-access-s9nlx" (OuterVolumeSpecName: "kube-api-access-s9nlx") pod "00cce627-9206-4775-a587-e3d8c44529b7" (UID: "00cce627-9206-4775-a587-e3d8c44529b7"). InnerVolumeSpecName "kube-api-access-s9nlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:03.128968 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.128854    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb57d469-efaa-43f2-974f-d25e023d3eae" (UID: "eb57d469-efaa-43f2-974f-d25e023d3eae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:40:03.128968 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.128872    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb57d469-efaa-43f2-974f-d25e023d3eae-kube-api-access-jswp5" (OuterVolumeSpecName: "kube-api-access-jswp5") pod "eb57d469-efaa-43f2-974f-d25e023d3eae" (UID: "eb57d469-efaa-43f2-974f-d25e023d3eae"). InnerVolumeSpecName "kube-api-access-jswp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:03.227472 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.227443    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswp5\" (UniqueName: \"kubernetes.io/projected/eb57d469-efaa-43f2-974f-d25e023d3eae-kube-api-access-jswp5\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:40:03.227472 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.227467    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cce627-9206-4775-a587-e3d8c44529b7-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:40:03.227472 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.227477    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:40:03.227693 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.227486    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb57d469-efaa-43f2-974f-d25e023d3eae-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:40:03.227693 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.227495    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb57d469-efaa-43f2-974f-d25e023d3eae-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:40:03.227693 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.227503    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nlx\" (UniqueName: \"kubernetes.io/projected/00cce627-9206-4775-a587-e3d8c44529b7-kube-api-access-s9nlx\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:40:03.380680 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.380646    2568 generic.go:356] "Generic (PLEG): container finished" podID="00cce627-9206-4775-a587-e3d8c44529b7" containerID="7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916" exitCode=0
Sep 10 17:40:03.380860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.380689    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"
Sep 10 17:40:03.380860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.380719    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx" event={"ID":"00cce627-9206-4775-a587-e3d8c44529b7","Type":"ContainerDied","Data":"7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916"}
Sep 10 17:40:03.380860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.380760    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx" event={"ID":"00cce627-9206-4775-a587-e3d8c44529b7","Type":"ContainerDied","Data":"a1674ead86df72bc41d7bafa24c009b6d1302760a5aedcdd4d408e02c923c7fe"}
Sep 10 17:40:03.380860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.380785    2568 scope.go:117] "RemoveContainer" containerID="7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916"
Sep 10 17:40:03.382073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.382051    2568 generic.go:356] "Generic (PLEG): container finished" podID="eb57d469-efaa-43f2-974f-d25e023d3eae" containerID="6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b" exitCode=0
Sep 10 17:40:03.382184 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.382073    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq" event={"ID":"eb57d469-efaa-43f2-974f-d25e023d3eae","Type":"ContainerDied","Data":"6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b"}
Sep 10 17:40:03.382184 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.382103    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"
Sep 10 17:40:03.382184 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.382115    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq" event={"ID":"eb57d469-efaa-43f2-974f-d25e023d3eae","Type":"ContainerDied","Data":"a7a0dda0a7e20122b8ba1bd30643d04cdfe62314e04b3ac4ded1f732a34919fc"}
Sep 10 17:40:03.389333 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.389287    2568 scope.go:117] "RemoveContainer" containerID="7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916"
Sep 10 17:40:03.389560 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:40:03.389540    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916\": container with ID starting with 7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916 not found: ID does not exist" containerID="7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916"
Sep 10 17:40:03.389644 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.389574    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916"} err="failed to get container status \"7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916\": rpc error: code = NotFound desc = could not find container \"7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916\": container with ID starting with 7b43e48e34a86bec328c3dbff035a8187ec8aa84d1e717b841857bc3f5396916 not found: ID does not exist"
Sep 10 17:40:03.389644 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.389596    2568 scope.go:117] "RemoveContainer" containerID="6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b"
Sep 10 17:40:03.397964 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.397951    2568 scope.go:117] "RemoveContainer" containerID="6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b"
Sep 10 17:40:03.398208 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:40:03.398189    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b\": container with ID starting with 6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b not found: ID does not exist" containerID="6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b"
Sep 10 17:40:03.398312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.398213    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b"} err="failed to get container status \"6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b\": rpc error: code = NotFound desc = could not find container \"6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b\": container with ID starting with 6d48bb5cf21e35cbd5cbdd83d9e30f90436815334f23e6dfbf65f9199cd7853b not found: ID does not exist"
Sep 10 17:40:03.409181 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.409159    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"]
Sep 10 17:40:03.416771 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.416752    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14b3e7-7fbcbcd448-vbksq"]
Sep 10 17:40:03.428363 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.428342    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"]
Sep 10 17:40:03.434369 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.434350    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14b3e7-58cc55d6cd-7qlfx"]
Sep 10 17:40:03.763789 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.763754    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00cce627-9206-4775-a587-e3d8c44529b7" path="/var/lib/kubelet/pods/00cce627-9206-4775-a587-e3d8c44529b7/volumes"
Sep 10 17:40:03.764169 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:03.764155    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb57d469-efaa-43f2-974f-d25e023d3eae" path="/var/lib/kubelet/pods/eb57d469-efaa-43f2-974f-d25e023d3eae/volumes"
Sep 10 17:40:34.935526 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.935489    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"]
Sep 10 17:40:34.936225 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.935945    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cce627-9206-4775-a587-e3d8c44529b7" containerName="payload-pre-processing"
Sep 10 17:40:34.936225 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.935958    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cce627-9206-4775-a587-e3d8c44529b7" containerName="payload-pre-processing"
Sep 10 17:40:34.936225 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.935988    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb57d469-efaa-43f2-974f-d25e023d3eae" containerName="maas-api"
Sep 10 17:40:34.936225 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.935994    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb57d469-efaa-43f2-974f-d25e023d3eae" containerName="maas-api"
Sep 10 17:40:34.936225 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.936059    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="00cce627-9206-4775-a587-e3d8c44529b7" containerName="payload-pre-processing"
Sep 10 17:40:34.936225 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.936073    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb57d469-efaa-43f2-974f-d25e023d3eae" containerName="maas-api"
Sep 10 17:40:34.939486 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.939466    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:34.942470 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.942445    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-aee2c26d-serving-cert\""
Sep 10 17:40:34.949294 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.949230    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"]
Sep 10 17:40:34.997420 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.997371    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4fb\" (UniqueName: \"kubernetes.io/projected/ac573764-9a84-4a1d-9052-e9715360d3bb-kube-api-access-7b4fb\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:34.997588 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.997432    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:34.997588 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.997453    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-metrics-certs\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:34.997588 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:34.997480    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac573764-9a84-4a1d-9052-e9715360d3bb-openshift-service-ca\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.098166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.098134    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4fb\" (UniqueName: \"kubernetes.io/projected/ac573764-9a84-4a1d-9052-e9715360d3bb-kube-api-access-7b4fb\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.098339 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.098182    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.098339 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:40:35.098279    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-serving-cert: secret "maas-api-e2e-mt-aee2c26d-serving-cert" not found
Sep 10 17:40:35.098339 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.098288    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-metrics-certs\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.098339 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.098331    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac573764-9a84-4a1d-9052-e9715360d3bb-openshift-service-ca\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.098502 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:40:35.098350    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls podName:ac573764-9a84-4a1d-9052-e9715360d3bb nodeName:}" failed. No retries permitted until 2026-09-10 17:40:35.598334153 +0000 UTC m=+1004.307928396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls") pod "maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw" (UID: "ac573764-9a84-4a1d-9052-e9715360d3bb") : secret "maas-api-e2e-mt-aee2c26d-serving-cert" not found
Sep 10 17:40:35.099102 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.099083    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac573764-9a84-4a1d-9052-e9715360d3bb-openshift-service-ca\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.100763 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.100742    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-metrics-certs\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.109653 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.109630    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4fb\" (UniqueName: \"kubernetes.io/projected/ac573764-9a84-4a1d-9052-e9715360d3bb-kube-api-access-7b4fb\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.592065 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.592028    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"]
Sep 10 17:40:35.596006 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.595986    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:35.598998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.598968    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-aee2c26d-dockercfg-4jtrj\""
Sep 10 17:40:35.599280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.599262    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-aee2c26d\""
Sep 10 17:40:35.603227 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.603196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.605868 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.605842    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls\") pod \"maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.613802 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.613773    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"]
Sep 10 17:40:35.615938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.615918    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"]
Sep 10 17:40:35.619380 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.619362    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:35.633717 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.633688    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"]
Sep 10 17:40:35.704621 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.704587    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtq55\" (UniqueName: \"kubernetes.io/projected/0fd157c5-168d-421f-9cdb-5bc58345d1a1-kube-api-access-xtq55\") pod \"payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9\" (UID: \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:35.704809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.704702    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd157c5-168d-421f-9cdb-5bc58345d1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9\" (UID: \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:35.704809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.704741    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-plugins-config-volume\") pod \"payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5\" (UID: \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\") " pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:35.704809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.704779    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zp85\" (UniqueName: \"kubernetes.io/projected/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-kube-api-access-2zp85\") pod \"payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5\" (UID: \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\") " pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:35.805330 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.805290    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd157c5-168d-421f-9cdb-5bc58345d1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9\" (UID: \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:35.805330 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.805330    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-plugins-config-volume\") pod \"payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5\" (UID: \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\") " pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:35.805547 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.805358    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zp85\" (UniqueName: \"kubernetes.io/projected/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-kube-api-access-2zp85\") pod \"payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5\" (UID: \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\") " pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:35.805547 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.805396    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtq55\" (UniqueName: \"kubernetes.io/projected/0fd157c5-168d-421f-9cdb-5bc58345d1a1-kube-api-access-xtq55\") pod \"payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9\" (UID: \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:35.806212 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.806187    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd157c5-168d-421f-9cdb-5bc58345d1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9\" (UID: \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:35.806294 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.806187    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-plugins-config-volume\") pod \"payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5\" (UID: \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\") " pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:35.816893 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.816870    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zp85\" (UniqueName: \"kubernetes.io/projected/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-kube-api-access-2zp85\") pod \"payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5\" (UID: \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\") " pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:35.817085 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.817063    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtq55\" (UniqueName: \"kubernetes.io/projected/0fd157c5-168d-421f-9cdb-5bc58345d1a1-kube-api-access-xtq55\") pod \"payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9\" (UID: \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:35.851871 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.851795    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:35.907454 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.907424    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:35.930425 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:35.930365    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:36.123953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.123931    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"]
Sep 10 17:40:36.125936 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:40:36.125913    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac573764_9a84_4a1d_9052_e9715360d3bb.slice/crio-2f2eb1657e8a126468951a5e150b287aeed508e61431bbdc4075c56fa03d3c41 WatchSource:0}: Error finding container 2f2eb1657e8a126468951a5e150b287aeed508e61431bbdc4075c56fa03d3c41: Status 404 returned error can't find the container with id 2f2eb1657e8a126468951a5e150b287aeed508e61431bbdc4075c56fa03d3c41
Sep 10 17:40:36.127227 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.127210    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:40:36.188938 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.188909    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"]
Sep 10 17:40:36.190213 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:40:36.190187    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd157c5_168d_421f_9cdb_5bc58345d1a1.slice/crio-4de9001abc0f218887d138770e020a7c40ecc4e4ca6bc454a4d9c42b0d650b30 WatchSource:0}: Error finding container 4de9001abc0f218887d138770e020a7c40ecc4e4ca6bc454a4d9c42b0d650b30: Status 404 returned error can't find the container with id 4de9001abc0f218887d138770e020a7c40ecc4e4ca6bc454a4d9c42b0d650b30
Sep 10 17:40:36.215969 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.215932    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"]
Sep 10 17:40:36.232521 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:40:36.232495    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a6460d_38bb_4e8f_9b01_1e008a38fa3f.slice/crio-37364c5f33c4d0aecfb620129694140ef487e1cc621be1fff3105e3369585ffb WatchSource:0}: Error finding container 37364c5f33c4d0aecfb620129694140ef487e1cc621be1fff3105e3369585ffb: Status 404 returned error can't find the container with id 37364c5f33c4d0aecfb620129694140ef487e1cc621be1fff3105e3369585ffb
Sep 10 17:40:36.555099 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.555062    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9" event={"ID":"0fd157c5-168d-421f-9cdb-5bc58345d1a1","Type":"ContainerStarted","Data":"a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47"}
Sep 10 17:40:36.555261 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.555132    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9" event={"ID":"0fd157c5-168d-421f-9cdb-5bc58345d1a1","Type":"ContainerStarted","Data":"4de9001abc0f218887d138770e020a7c40ecc4e4ca6bc454a4d9c42b0d650b30"}
Sep 10 17:40:36.555261 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.555180    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:36.556487 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.556463    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw" event={"ID":"ac573764-9a84-4a1d-9052-e9715360d3bb","Type":"ContainerStarted","Data":"2f2eb1657e8a126468951a5e150b287aeed508e61431bbdc4075c56fa03d3c41"}
Sep 10 17:40:36.557997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.557975    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5" event={"ID":"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f","Type":"ContainerStarted","Data":"f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e"}
Sep 10 17:40:36.558104 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.558022    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5" event={"ID":"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f","Type":"ContainerStarted","Data":"37364c5f33c4d0aecfb620129694140ef487e1cc621be1fff3105e3369585ffb"}
Sep 10 17:40:36.558104 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.558049    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:36.588646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.588597    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9" podStartSLOduration=1.588580771 podStartE2EDuration="1.588580771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:40:36.587016445 +0000 UTC m=+1005.296610712" watchObservedRunningTime="2026-09-10 17:40:36.588580771 +0000 UTC m=+1005.298175034"
Sep 10 17:40:36.618359 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:36.618306    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5" podStartSLOduration=1.61828971 podStartE2EDuration="1.61828971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:40:36.617012251 +0000 UTC m=+1005.326606530" watchObservedRunningTime="2026-09-10 17:40:36.61828971 +0000 UTC m=+1005.327884004"
Sep 10 17:40:37.572678 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:37.572644    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw" event={"ID":"ac573764-9a84-4a1d-9052-e9715360d3bb","Type":"ContainerStarted","Data":"26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae"}
Sep 10 17:40:37.573075 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:37.572738    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:37.592288 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:37.592219    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw" podStartSLOduration=3.195036579 podStartE2EDuration="3.592204552s" totalImagesPullingTime="397.167973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:40:37.590335473 +0000 UTC m=+1006.299929733" watchObservedRunningTime="2026-09-10 17:40:37.592204552 +0000 UTC m=+1006.301798818"
Sep 10 17:40:43.583840 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:43.583806    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:40:47.574852 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:47.574821    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:40:47.575297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:47.575280    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:40:59.808022 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.807982    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"]
Sep 10 17:40:59.813037 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.813014    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:40:59.822334 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.822305    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-085b3b\"/\"kube-root-ca.crt\""
Sep 10 17:40:59.823756 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.822504    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-085b3b\"/\"noten-model-085b3b-kserve-self-signed-certs\""
Sep 10 17:40:59.823756 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.823563    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-085b3b\"/\"default-dockercfg-2kdwz\""
Sep 10 17:40:59.823756 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.822585    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-085b3b\"/\"openshift-service-ca.crt\""
Sep 10 17:40:59.827856 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.827830    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"]
Sep 10 17:40:59.924400 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.924369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-model-cache\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:40:59.924581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.924438    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tmp-dir\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:40:59.924581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.924463    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf4mw\" (UniqueName: \"kubernetes.io/projected/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-kube-api-access-xf4mw\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:40:59.924581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.924490    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tls-certs\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:40:59.924581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.924534    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-home\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:40:59.924581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:40:59.924561    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-dshm\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.025630 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.025599    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tmp-dir\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.025826 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.025638    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf4mw\" (UniqueName: \"kubernetes.io/projected/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-kube-api-access-xf4mw\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.025826 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.025667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tls-certs\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.025826 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.025701    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-home\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.026004 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.025845    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-dshm\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.026004 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.025965    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-model-cache\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.026112 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.026072    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tmp-dir\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.026151 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.026113    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-home\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.026193 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.026183    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-model-cache\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.027955 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.027935    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-dshm\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.028134 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.028118    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tls-certs\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.034348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.034323    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf4mw\" (UniqueName: \"kubernetes.io/projected/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-kube-api-access-xf4mw\") pod \"noten-model-085b3b-kserve-78784cb54c-75tf6\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") " pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.129831 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.129746    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:00.401726 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.401698    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"]
Sep 10 17:41:00.403587 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:41:00.403558    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d985d5b_2bab_445e_bdb4_f4d37cdd9231.slice/crio-ded2aaf92b96ea566ac9debc65c130e682471665d8eb03088ada04668e9d3769 WatchSource:0}: Error finding container ded2aaf92b96ea566ac9debc65c130e682471665d8eb03088ada04668e9d3769: Status 404 returned error can't find the container with id ded2aaf92b96ea566ac9debc65c130e682471665d8eb03088ada04668e9d3769
Sep 10 17:41:00.697515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.697411    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6" event={"ID":"9d985d5b-2bab-445e-bdb4-f4d37cdd9231","Type":"ContainerStarted","Data":"ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884"}
Sep 10 17:41:00.697515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.697453    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6" event={"ID":"9d985d5b-2bab-445e-bdb4-f4d37cdd9231","Type":"ContainerStarted","Data":"ded2aaf92b96ea566ac9debc65c130e682471665d8eb03088ada04668e9d3769"}
Sep 10 17:41:00.716598 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:00.716547    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6" podStartSLOduration=1.716528346 podStartE2EDuration="1.716528346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:00.716080143 +0000 UTC m=+1029.425674411" watchObservedRunningTime="2026-09-10 17:41:00.716528346 +0000 UTC m=+1029.426122648"
Sep 10 17:41:10.130416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:10.130378    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:10.130929 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:10.130460    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:10.143685 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:10.143661    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:10.765671 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:10.765639    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:13.715247 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:13.715197    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"]
Sep 10 17:41:13.715611 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:13.715411    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5" podUID="a9815a30-c0f0-4280-a046-318d9b7fd340" containerName="maas-api" containerID="cri-o://6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e" gracePeriod=30
Sep 10 17:41:13.976157 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:13.976090    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:41:14.051853 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.051820    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls\") pod \"a9815a30-c0f0-4280-a046-318d9b7fd340\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") "
Sep 10 17:41:14.052031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.051908    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-metrics-certs\") pod \"a9815a30-c0f0-4280-a046-318d9b7fd340\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") "
Sep 10 17:41:14.052031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.051964    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9815a30-c0f0-4280-a046-318d9b7fd340-openshift-service-ca\") pod \"a9815a30-c0f0-4280-a046-318d9b7fd340\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") "
Sep 10 17:41:14.052031 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.051981    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8kxw\" (UniqueName: \"kubernetes.io/projected/a9815a30-c0f0-4280-a046-318d9b7fd340-kube-api-access-m8kxw\") pod \"a9815a30-c0f0-4280-a046-318d9b7fd340\" (UID: \"a9815a30-c0f0-4280-a046-318d9b7fd340\") "
Sep 10 17:41:14.052377 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.052347    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9815a30-c0f0-4280-a046-318d9b7fd340-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9815a30-c0f0-4280-a046-318d9b7fd340" (UID: "a9815a30-c0f0-4280-a046-318d9b7fd340"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:14.053998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.053973    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9815a30-c0f0-4280-a046-318d9b7fd340-kube-api-access-m8kxw" (OuterVolumeSpecName: "kube-api-access-m8kxw") pod "a9815a30-c0f0-4280-a046-318d9b7fd340" (UID: "a9815a30-c0f0-4280-a046-318d9b7fd340"). InnerVolumeSpecName "kube-api-access-m8kxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:14.053998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.053986    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9815a30-c0f0-4280-a046-318d9b7fd340" (UID: "a9815a30-c0f0-4280-a046-318d9b7fd340"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:14.054145 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.054010    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a9815a30-c0f0-4280-a046-318d9b7fd340" (UID: "a9815a30-c0f0-4280-a046-318d9b7fd340"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:14.153594 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.153555    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:14.153594 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.153592    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9815a30-c0f0-4280-a046-318d9b7fd340-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:14.153783 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.153607    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8kxw\" (UniqueName: \"kubernetes.io/projected/a9815a30-c0f0-4280-a046-318d9b7fd340-kube-api-access-m8kxw\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:14.153783 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.153622    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9815a30-c0f0-4280-a046-318d9b7fd340-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:14.776256 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.776199    2568 generic.go:356] "Generic (PLEG): container finished" podID="a9815a30-c0f0-4280-a046-318d9b7fd340" containerID="6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e" exitCode=0
Sep 10 17:41:14.776256 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.776249    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"
Sep 10 17:41:14.776786 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.776268    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5" event={"ID":"a9815a30-c0f0-4280-a046-318d9b7fd340","Type":"ContainerDied","Data":"6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e"}
Sep 10 17:41:14.776786 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.776334    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5" event={"ID":"a9815a30-c0f0-4280-a046-318d9b7fd340","Type":"ContainerDied","Data":"fe33a97fc93d3e0d76ff029a5de8202b9a0f53753042eb90a479103a615006c5"}
Sep 10 17:41:14.776786 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.776367    2568 scope.go:117] "RemoveContainer" containerID="6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e"
Sep 10 17:41:14.785578 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.785560    2568 scope.go:117] "RemoveContainer" containerID="6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e"
Sep 10 17:41:14.785833 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:14.785817    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e\": container with ID starting with 6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e not found: ID does not exist" containerID="6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e"
Sep 10 17:41:14.785878 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.785841    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e"} err="failed to get container status \"6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e\": rpc error: code = NotFound desc = could not find container \"6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e\": container with ID starting with 6d07af05ec960951afc0d44a1a763818316aae3b6a55ee771bf27e0f53ae894e not found: ID does not exist"
Sep 10 17:41:14.802525 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.802489    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"]
Sep 10 17:41:14.804227 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:14.804204    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d25fa-66b7658c69-cjlp5"]
Sep 10 17:41:15.763481 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:15.763443    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9815a30-c0f0-4280-a046-318d9b7fd340" path="/var/lib/kubelet/pods/a9815a30-c0f0-4280-a046-318d9b7fd340/volumes"
Sep 10 17:41:17.286190 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.286152    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"]
Sep 10 17:41:17.286687 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.286411    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw" podUID="ac573764-9a84-4a1d-9052-e9715360d3bb" containerName="maas-api" containerID="cri-o://26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae" gracePeriod=30
Sep 10 17:41:17.341246 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.341190    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"]
Sep 10 17:41:17.341504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.341478    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5" podUID="c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" containerName="payload-processing" containerID="cri-o://f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e" gracePeriod=30
Sep 10 17:41:17.355812 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.354692    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"]
Sep 10 17:41:17.355812 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.354915    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9" podUID="0fd157c5-168d-421f-9cdb-5bc58345d1a1" containerName="payload-pre-processing" containerID="cri-o://a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47" gracePeriod=30
Sep 10 17:41:17.573714 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.573672    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9" podUID="0fd157c5-168d-421f-9cdb-5bc58345d1a1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.68:9004: connect: connection refused"
Sep 10 17:41:17.615368 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.615340    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:41:17.646953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.646928    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:41:17.656089 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.656065    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:41:17.690048 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690018    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-metrics-certs\") pod \"ac573764-9a84-4a1d-9052-e9715360d3bb\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") "
Sep 10 17:41:17.690048 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690053    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b4fb\" (UniqueName: \"kubernetes.io/projected/ac573764-9a84-4a1d-9052-e9715360d3bb-kube-api-access-7b4fb\") pod \"ac573764-9a84-4a1d-9052-e9715360d3bb\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") "
Sep 10 17:41:17.690321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690079    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd157c5-168d-421f-9cdb-5bc58345d1a1-plugins-config-volume\") pod \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\" (UID: \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\") "
Sep 10 17:41:17.690321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690094    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zp85\" (UniqueName: \"kubernetes.io/projected/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-kube-api-access-2zp85\") pod \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\" (UID: \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\") "
Sep 10 17:41:17.690321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690143    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac573764-9a84-4a1d-9052-e9715360d3bb-openshift-service-ca\") pod \"ac573764-9a84-4a1d-9052-e9715360d3bb\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") "
Sep 10 17:41:17.690321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690175    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtq55\" (UniqueName: \"kubernetes.io/projected/0fd157c5-168d-421f-9cdb-5bc58345d1a1-kube-api-access-xtq55\") pod \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\" (UID: \"0fd157c5-168d-421f-9cdb-5bc58345d1a1\") "
Sep 10 17:41:17.690321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690202    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls\") pod \"ac573764-9a84-4a1d-9052-e9715360d3bb\" (UID: \"ac573764-9a84-4a1d-9052-e9715360d3bb\") "
Sep 10 17:41:17.690321 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690255    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-plugins-config-volume\") pod \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\" (UID: \"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f\") "
Sep 10 17:41:17.690660 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690473    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd157c5-168d-421f-9cdb-5bc58345d1a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fd157c5-168d-421f-9cdb-5bc58345d1a1" (UID: "0fd157c5-168d-421f-9cdb-5bc58345d1a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:17.690660 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690583    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd157c5-168d-421f-9cdb-5bc58345d1a1-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.690814 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690791    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" (UID: "c0a6460d-38bb-4e8f-9b01-1e008a38fa3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:17.690884 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.690825    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac573764-9a84-4a1d-9052-e9715360d3bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac573764-9a84-4a1d-9052-e9715360d3bb" (UID: "ac573764-9a84-4a1d-9052-e9715360d3bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:17.692887 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.692853    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-kube-api-access-2zp85" (OuterVolumeSpecName: "kube-api-access-2zp85") pod "c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" (UID: "c0a6460d-38bb-4e8f-9b01-1e008a38fa3f"). InnerVolumeSpecName "kube-api-access-2zp85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:17.692998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.692897    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac573764-9a84-4a1d-9052-e9715360d3bb" (UID: "ac573764-9a84-4a1d-9052-e9715360d3bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:17.692998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.692934    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac573764-9a84-4a1d-9052-e9715360d3bb-kube-api-access-7b4fb" (OuterVolumeSpecName: "kube-api-access-7b4fb") pod "ac573764-9a84-4a1d-9052-e9715360d3bb" (UID: "ac573764-9a84-4a1d-9052-e9715360d3bb"). InnerVolumeSpecName "kube-api-access-7b4fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:17.692998 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.692979    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd157c5-168d-421f-9cdb-5bc58345d1a1-kube-api-access-xtq55" (OuterVolumeSpecName: "kube-api-access-xtq55") pod "0fd157c5-168d-421f-9cdb-5bc58345d1a1" (UID: "0fd157c5-168d-421f-9cdb-5bc58345d1a1"). InnerVolumeSpecName "kube-api-access-xtq55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:17.693112 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.693002    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac573764-9a84-4a1d-9052-e9715360d3bb" (UID: "ac573764-9a84-4a1d-9052-e9715360d3bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:17.791478 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.791452    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.791478 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.791478    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b4fb\" (UniqueName: \"kubernetes.io/projected/ac573764-9a84-4a1d-9052-e9715360d3bb-kube-api-access-7b4fb\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.791650 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.791489    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zp85\" (UniqueName: \"kubernetes.io/projected/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-kube-api-access-2zp85\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.791650 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.791500    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac573764-9a84-4a1d-9052-e9715360d3bb-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.791650 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.791512    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtq55\" (UniqueName: \"kubernetes.io/projected/0fd157c5-168d-421f-9cdb-5bc58345d1a1-kube-api-access-xtq55\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.791650 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.791522    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac573764-9a84-4a1d-9052-e9715360d3bb-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.791650 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.791535    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:17.792515 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.792490    2568 generic.go:356] "Generic (PLEG): container finished" podID="0fd157c5-168d-421f-9cdb-5bc58345d1a1" containerID="a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47" exitCode=0
Sep 10 17:41:17.792601 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.792535    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"
Sep 10 17:41:17.792601 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.792561    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9" event={"ID":"0fd157c5-168d-421f-9cdb-5bc58345d1a1","Type":"ContainerDied","Data":"a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47"}
Sep 10 17:41:17.792706 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.792611    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9" event={"ID":"0fd157c5-168d-421f-9cdb-5bc58345d1a1","Type":"ContainerDied","Data":"4de9001abc0f218887d138770e020a7c40ecc4e4ca6bc454a4d9c42b0d650b30"}
Sep 10 17:41:17.792706 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.792644    2568 scope.go:117] "RemoveContainer" containerID="a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47"
Sep 10 17:41:17.793961 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.793940    2568 generic.go:356] "Generic (PLEG): container finished" podID="ac573764-9a84-4a1d-9052-e9715360d3bb" containerID="26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae" exitCode=0
Sep 10 17:41:17.794050 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.793990    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"
Sep 10 17:41:17.794050 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.794016    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw" event={"ID":"ac573764-9a84-4a1d-9052-e9715360d3bb","Type":"ContainerDied","Data":"26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae"}
Sep 10 17:41:17.794206 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.794069    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw" event={"ID":"ac573764-9a84-4a1d-9052-e9715360d3bb","Type":"ContainerDied","Data":"2f2eb1657e8a126468951a5e150b287aeed508e61431bbdc4075c56fa03d3c41"}
Sep 10 17:41:17.795642 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.795620    2568 generic.go:356] "Generic (PLEG): container finished" podID="c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" containerID="f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e" exitCode=0
Sep 10 17:41:17.795751 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.795647    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"
Sep 10 17:41:17.795751 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.795667    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5" event={"ID":"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f","Type":"ContainerDied","Data":"f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e"}
Sep 10 17:41:17.795751 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.795704    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5" event={"ID":"c0a6460d-38bb-4e8f-9b01-1e008a38fa3f","Type":"ContainerDied","Data":"37364c5f33c4d0aecfb620129694140ef487e1cc621be1fff3105e3369585ffb"}
Sep 10 17:41:17.802160 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.802142    2568 scope.go:117] "RemoveContainer" containerID="a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47"
Sep 10 17:41:17.802460 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:17.802440    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47\": container with ID starting with a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47 not found: ID does not exist" containerID="a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47"
Sep 10 17:41:17.802521 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.802469    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47"} err="failed to get container status \"a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47\": rpc error: code = NotFound desc = could not find container \"a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47\": container with ID starting with a259ca995a3cb438ea707f2f2d0576c78ad8049115acc7f5f25385d27ef2ce47 not found: ID does not exist"
Sep 10 17:41:17.802521 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.802487    2568 scope.go:117] "RemoveContainer" containerID="26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae"
Sep 10 17:41:17.810652 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.810634    2568 scope.go:117] "RemoveContainer" containerID="26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae"
Sep 10 17:41:17.810920 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:17.810902    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae\": container with ID starting with 26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae not found: ID does not exist" containerID="26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae"
Sep 10 17:41:17.810970 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.810926    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae"} err="failed to get container status \"26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae\": rpc error: code = NotFound desc = could not find container \"26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae\": container with ID starting with 26b734e9df9df17d3d2759013b0f591f2242e50830c1af570e2d5aa55fe865ae not found: ID does not exist"
Sep 10 17:41:17.810970 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.810941    2568 scope.go:117] "RemoveContainer" containerID="f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e"
Sep 10 17:41:17.815166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.815145    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"]
Sep 10 17:41:17.818667 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.818647    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee2c26d-6fd9f94bb6-mjhh9"]
Sep 10 17:41:17.818992 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.818976    2568 scope.go:117] "RemoveContainer" containerID="f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e"
Sep 10 17:41:17.819290 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:17.819272    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e\": container with ID starting with f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e not found: ID does not exist" containerID="f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e"
Sep 10 17:41:17.819383 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.819294    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e"} err="failed to get container status \"f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e\": rpc error: code = NotFound desc = could not find container \"f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e\": container with ID starting with f3bcd8a03448be2ea9e6cac605836db7839f0f6708ccea9564383d3603ea7e6e not found: ID does not exist"
Sep 10 17:41:17.832893 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.832874    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"]
Sep 10 17:41:17.836689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.836669    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5"]
Sep 10 17:41:17.848368 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.848343    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"]
Sep 10 17:41:17.851696 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:17.851673    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee2c26d-559df6666f-pvhxw"]
Sep 10 17:41:18.573882 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:18.573792    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-aee2c26d-6b9f644f4f-45pm5" podUID="c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.69:9004: i/o timeout"
Sep 10 17:41:19.289056 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:19.289021    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"]
Sep 10 17:41:19.763869 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:19.763788    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd157c5-168d-421f-9cdb-5bc58345d1a1" path="/var/lib/kubelet/pods/0fd157c5-168d-421f-9cdb-5bc58345d1a1/volumes"
Sep 10 17:41:19.764218 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:19.764169    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac573764-9a84-4a1d-9052-e9715360d3bb" path="/var/lib/kubelet/pods/ac573764-9a84-4a1d-9052-e9715360d3bb/volumes"
Sep 10 17:41:19.764502 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:19.764488    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" path="/var/lib/kubelet/pods/c0a6460d-38bb-4e8f-9b01-1e008a38fa3f/volumes"
Sep 10 17:41:31.556683 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.556651    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"]
Sep 10 17:41:31.557111 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.556852    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z" podUID="4f2b010b-fbd5-4059-a14d-0a5a52dd4354" containerName="maas-api" containerID="cri-o://bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4" gracePeriod=30
Sep 10 17:41:31.615904 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.615754    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"]
Sep 10 17:41:31.616093 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.616016    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls" podUID="439ea1b8-0bc2-438b-a259-8ce664f5fc05" containerName="payload-pre-processing" containerID="cri-o://a9ab03ade42a362cb8392de35cff96e19411e0bbc428e92bd2a0ee208eff287c" gracePeriod=30
Sep 10 17:41:31.867974 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.867947    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:41:31.869959 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.869934    2568 generic.go:356] "Generic (PLEG): container finished" podID="4f2b010b-fbd5-4059-a14d-0a5a52dd4354" containerID="bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4" exitCode=0
Sep 10 17:41:31.870079 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.869977    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"
Sep 10 17:41:31.870079 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.870017    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z" event={"ID":"4f2b010b-fbd5-4059-a14d-0a5a52dd4354","Type":"ContainerDied","Data":"bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4"}
Sep 10 17:41:31.870192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.870111    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z" event={"ID":"4f2b010b-fbd5-4059-a14d-0a5a52dd4354","Type":"ContainerDied","Data":"044899384f00541699acabf1f0e82c39b6f2a7bae26bb19aad3f66fad13f969e"}
Sep 10 17:41:31.870192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.870144    2568 scope.go:117] "RemoveContainer" containerID="bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4"
Sep 10 17:41:31.871920 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.871901    2568 generic.go:356] "Generic (PLEG): container finished" podID="439ea1b8-0bc2-438b-a259-8ce664f5fc05" containerID="a9ab03ade42a362cb8392de35cff96e19411e0bbc428e92bd2a0ee208eff287c" exitCode=0
Sep 10 17:41:31.872026 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.871945    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls" event={"ID":"439ea1b8-0bc2-438b-a259-8ce664f5fc05","Type":"ContainerDied","Data":"a9ab03ade42a362cb8392de35cff96e19411e0bbc428e92bd2a0ee208eff287c"}
Sep 10 17:41:31.878928 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.878911    2568 scope.go:117] "RemoveContainer" containerID="bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4"
Sep 10 17:41:31.879259 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:31.879207    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4\": container with ID starting with bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4 not found: ID does not exist" containerID="bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4"
Sep 10 17:41:31.879331 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.879274    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4"} err="failed to get container status \"bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4\": rpc error: code = NotFound desc = could not find container \"bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4\": container with ID starting with bf251c2d9d1db79ec6aa95e3a8cf096d7cd2bdb62ad5e4ff1e6ddae397dda1d4 not found: ID does not exist"
Sep 10 17:41:31.897642 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:31.897618    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:41:32.025198 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.025170    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlssn\" (UniqueName: \"kubernetes.io/projected/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-kube-api-access-vlssn\") pod \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") "
Sep 10 17:41:32.025198 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.025205    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls\") pod \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") "
Sep 10 17:41:32.025460 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.025264    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-openshift-service-ca\") pod \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") "
Sep 10 17:41:32.025460 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.025347    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-metrics-certs\") pod \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\" (UID: \"4f2b010b-fbd5-4059-a14d-0a5a52dd4354\") "
Sep 10 17:41:32.025460 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.025376    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439ea1b8-0bc2-438b-a259-8ce664f5fc05-plugins-config-volume\") pod \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\" (UID: \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\") "
Sep 10 17:41:32.025460 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.025393    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4vcx\" (UniqueName: \"kubernetes.io/projected/439ea1b8-0bc2-438b-a259-8ce664f5fc05-kube-api-access-j4vcx\") pod \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\" (UID: \"439ea1b8-0bc2-438b-a259-8ce664f5fc05\") "
Sep 10 17:41:32.025768 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.025726    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f2b010b-fbd5-4059-a14d-0a5a52dd4354" (UID: "4f2b010b-fbd5-4059-a14d-0a5a52dd4354"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:32.025843 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.025817    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439ea1b8-0bc2-438b-a259-8ce664f5fc05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "439ea1b8-0bc2-438b-a259-8ce664f5fc05" (UID: "439ea1b8-0bc2-438b-a259-8ce664f5fc05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:32.027572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.027541    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f2b010b-fbd5-4059-a14d-0a5a52dd4354" (UID: "4f2b010b-fbd5-4059-a14d-0a5a52dd4354"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:32.027688 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.027638    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439ea1b8-0bc2-438b-a259-8ce664f5fc05-kube-api-access-j4vcx" (OuterVolumeSpecName: "kube-api-access-j4vcx") pod "439ea1b8-0bc2-438b-a259-8ce664f5fc05" (UID: "439ea1b8-0bc2-438b-a259-8ce664f5fc05"). InnerVolumeSpecName "kube-api-access-j4vcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:32.027836 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.027821    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-kube-api-access-vlssn" (OuterVolumeSpecName: "kube-api-access-vlssn") pod "4f2b010b-fbd5-4059-a14d-0a5a52dd4354" (UID: "4f2b010b-fbd5-4059-a14d-0a5a52dd4354"). InnerVolumeSpecName "kube-api-access-vlssn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:32.027875 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.027822    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4f2b010b-fbd5-4059-a14d-0a5a52dd4354" (UID: "4f2b010b-fbd5-4059-a14d-0a5a52dd4354"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:32.126796 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.126717    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:32.126796 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.126743    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:32.126796 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.126752    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439ea1b8-0bc2-438b-a259-8ce664f5fc05-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:32.126796 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.126761    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4vcx\" (UniqueName: \"kubernetes.io/projected/439ea1b8-0bc2-438b-a259-8ce664f5fc05-kube-api-access-j4vcx\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:32.126796 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.126770    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlssn\" (UniqueName: \"kubernetes.io/projected/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-kube-api-access-vlssn\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:32.126796 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.126788    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2b010b-fbd5-4059-a14d-0a5a52dd4354-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:32.192593 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.192561    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"]
Sep 10 17:41:32.196183 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.196154    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f1093-5c8646d57f-npb6z"]
Sep 10 17:41:32.879494 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.879409    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"
Sep 10 17:41:32.879494 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.879414    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls" event={"ID":"439ea1b8-0bc2-438b-a259-8ce664f5fc05","Type":"ContainerDied","Data":"c3a10b5a1bbb83225a6a7eb374485646d97b138ad74a0bb5c6357af18b2a3b1b"}
Sep 10 17:41:32.879995 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.879550    2568 scope.go:117] "RemoveContainer" containerID="a9ab03ade42a362cb8392de35cff96e19411e0bbc428e92bd2a0ee208eff287c"
Sep 10 17:41:32.901982 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.901961    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"]
Sep 10 17:41:32.905893 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:32.905867    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2f1093-64d787b846-c9bls"]
Sep 10 17:41:33.764777 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:33.764741    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="439ea1b8-0bc2-438b-a259-8ce664f5fc05" path="/var/lib/kubelet/pods/439ea1b8-0bc2-438b-a259-8ce664f5fc05/volumes"
Sep 10 17:41:33.765138 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:33.765125    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f2b010b-fbd5-4059-a14d-0a5a52dd4354" path="/var/lib/kubelet/pods/4f2b010b-fbd5-4059-a14d-0a5a52dd4354/volumes"
Sep 10 17:41:34.309096 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.309048    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6" podUID="9d985d5b-2bab-445e-bdb4-f4d37cdd9231" containerName="main" containerID="cri-o://ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884" gracePeriod=45
Sep 10 17:41:34.569979 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.569923    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:34.645738 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.645697    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-model-cache\") pod \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") "
Sep 10 17:41:34.645738 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.645733    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tmp-dir\") pod \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") "
Sep 10 17:41:34.645978 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.645764    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-home\") pod \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") "
Sep 10 17:41:34.645978 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.645796    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-dshm\") pod \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") "
Sep 10 17:41:34.645978 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.645857    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tls-certs\") pod \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") "
Sep 10 17:41:34.645978 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.645885    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf4mw\" (UniqueName: \"kubernetes.io/projected/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-kube-api-access-xf4mw\") pod \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\" (UID: \"9d985d5b-2bab-445e-bdb4-f4d37cdd9231\") "
Sep 10 17:41:34.646189 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.646076    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-home" (OuterVolumeSpecName: "home") pod "9d985d5b-2bab-445e-bdb4-f4d37cdd9231" (UID: "9d985d5b-2bab-445e-bdb4-f4d37cdd9231"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:34.646189 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.646106    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d985d5b-2bab-445e-bdb4-f4d37cdd9231" (UID: "9d985d5b-2bab-445e-bdb4-f4d37cdd9231"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:34.646319 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.646192    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:34.646319 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.646209    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:34.646432 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.646399    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d985d5b-2bab-445e-bdb4-f4d37cdd9231" (UID: "9d985d5b-2bab-445e-bdb4-f4d37cdd9231"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:34.648354 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.648317    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d985d5b-2bab-445e-bdb4-f4d37cdd9231" (UID: "9d985d5b-2bab-445e-bdb4-f4d37cdd9231"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:41:34.648354 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.648332    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-kube-api-access-xf4mw" (OuterVolumeSpecName: "kube-api-access-xf4mw") pod "9d985d5b-2bab-445e-bdb4-f4d37cdd9231" (UID: "9d985d5b-2bab-445e-bdb4-f4d37cdd9231"). InnerVolumeSpecName "kube-api-access-xf4mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:34.648547 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.648406    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-dshm" (OuterVolumeSpecName: "dshm") pod "9d985d5b-2bab-445e-bdb4-f4d37cdd9231" (UID: "9d985d5b-2bab-445e-bdb4-f4d37cdd9231"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:41:34.747559 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.747517    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:34.747559 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.747554    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf4mw\" (UniqueName: \"kubernetes.io/projected/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-kube-api-access-xf4mw\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:34.747805 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.747569    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:34.747805 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.747583    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d985d5b-2bab-445e-bdb4-f4d37cdd9231-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:41:34.891918 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.891826    2568 generic.go:356] "Generic (PLEG): container finished" podID="9d985d5b-2bab-445e-bdb4-f4d37cdd9231" containerID="ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884" exitCode=0
Sep 10 17:41:34.891918 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.891872    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"
Sep 10 17:41:34.892124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.891919    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6" event={"ID":"9d985d5b-2bab-445e-bdb4-f4d37cdd9231","Type":"ContainerDied","Data":"ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884"}
Sep 10 17:41:34.892124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.891965    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6" event={"ID":"9d985d5b-2bab-445e-bdb4-f4d37cdd9231","Type":"ContainerDied","Data":"ded2aaf92b96ea566ac9debc65c130e682471665d8eb03088ada04668e9d3769"}
Sep 10 17:41:34.892124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.891983    2568 scope.go:117] "RemoveContainer" containerID="ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884"
Sep 10 17:41:34.903664 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.903389    2568 scope.go:117] "RemoveContainer" containerID="ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884"
Sep 10 17:41:34.903900 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:34.903771    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884\": container with ID starting with ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884 not found: ID does not exist" containerID="ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884"
Sep 10 17:41:34.903900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.903811    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884"} err="failed to get container status \"ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884\": rpc error: code = NotFound desc = could not find container \"ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884\": container with ID starting with ebe99f706cb1ee8ba8e7e2e07463d0309fcc99bd8a15f2e19ff506fa74d8f884 not found: ID does not exist"
Sep 10 17:41:34.919211 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.919183    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"]
Sep 10 17:41:34.924745 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:34.924720    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-085b3b/noten-model-085b3b-kserve-78784cb54c-75tf6"]
Sep 10 17:41:35.762887 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:35.762847    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d985d5b-2bab-445e-bdb4-f4d37cdd9231" path="/var/lib/kubelet/pods/9d985d5b-2bab-445e-bdb4-f4d37cdd9231/volumes"
Sep 10 17:41:42.015719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.015683    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"]
Sep 10 17:41:42.016312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016226    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="439ea1b8-0bc2-438b-a259-8ce664f5fc05" containerName="payload-pre-processing"
Sep 10 17:41:42.016312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016265    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="439ea1b8-0bc2-438b-a259-8ce664f5fc05" containerName="payload-pre-processing"
Sep 10 17:41:42.016312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016284    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f2b010b-fbd5-4059-a14d-0a5a52dd4354" containerName="maas-api"
Sep 10 17:41:42.016312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016293    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f2b010b-fbd5-4059-a14d-0a5a52dd4354" containerName="maas-api"
Sep 10 17:41:42.016312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016307    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac573764-9a84-4a1d-9052-e9715360d3bb" containerName="maas-api"
Sep 10 17:41:42.016312 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016315    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac573764-9a84-4a1d-9052-e9715360d3bb" containerName="maas-api"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016324    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d985d5b-2bab-445e-bdb4-f4d37cdd9231" containerName="main"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016332    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d985d5b-2bab-445e-bdb4-f4d37cdd9231" containerName="main"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016344    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd157c5-168d-421f-9cdb-5bc58345d1a1" containerName="payload-pre-processing"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016352    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd157c5-168d-421f-9cdb-5bc58345d1a1" containerName="payload-pre-processing"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016370    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9815a30-c0f0-4280-a046-318d9b7fd340" containerName="maas-api"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016379    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9815a30-c0f0-4280-a046-318d9b7fd340" containerName="maas-api"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016393    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" containerName="payload-processing"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016401    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" containerName="payload-processing"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016495    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd157c5-168d-421f-9cdb-5bc58345d1a1" containerName="payload-pre-processing"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016507    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f2b010b-fbd5-4059-a14d-0a5a52dd4354" containerName="maas-api"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016520    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9815a30-c0f0-4280-a046-318d9b7fd340" containerName="maas-api"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016530    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac573764-9a84-4a1d-9052-e9715360d3bb" containerName="maas-api"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016541    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a6460d-38bb-4e8f-9b01-1e008a38fa3f" containerName="payload-processing"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016550    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d985d5b-2bab-445e-bdb4-f4d37cdd9231" containerName="main"
Sep 10 17:41:42.016646 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.016559    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="439ea1b8-0bc2-438b-a259-8ce664f5fc05" containerName="payload-pre-processing"
Sep 10 17:41:42.021367 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.021344    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.024201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.024178    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c990a5-serving-cert\""
Sep 10 17:41:42.031923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.031895    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"]
Sep 10 17:41:42.113599 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.113562    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.113599 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.113605    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8d8m\" (UniqueName: \"kubernetes.io/projected/c0121041-22a8-4c21-8ad7-da8f9edb32b1-kube-api-access-s8d8m\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.113799 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.113640    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.113799 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.113695    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0121041-22a8-4c21-8ad7-da8f9edb32b1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.214922 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.214882    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.215113 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.214929    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8d8m\" (UniqueName: \"kubernetes.io/projected/c0121041-22a8-4c21-8ad7-da8f9edb32b1-kube-api-access-s8d8m\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.215113 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.214978    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.215113 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.215024    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0121041-22a8-4c21-8ad7-da8f9edb32b1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.215113 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:42.215041    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-serving-cert: secret "maas-api-e2e-shared-a-w2-c990a5-serving-cert" not found
Sep 10 17:41:42.215113 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:42.215117    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls podName:c0121041-22a8-4c21-8ad7-da8f9edb32b1 nodeName:}" failed. No retries permitted until 2026-09-10 17:41:42.715097009 +0000 UTC m=+1071.424691262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk" (UID: "c0121041-22a8-4c21-8ad7-da8f9edb32b1") : secret "maas-api-e2e-shared-a-w2-c990a5-serving-cert" not found
Sep 10 17:41:42.215729 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.215710    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0121041-22a8-4c21-8ad7-da8f9edb32b1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.217484 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.217465    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.232626 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.232599    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8d8m\" (UniqueName: \"kubernetes.io/projected/c0121041-22a8-4c21-8ad7-da8f9edb32b1-kube-api-access-s8d8m\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.720125 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.720091    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:42.720305 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:42.720254    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-serving-cert: secret "maas-api-e2e-shared-a-w2-c990a5-serving-cert" not found
Sep 10 17:41:42.720354 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:42.720331    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls podName:c0121041-22a8-4c21-8ad7-da8f9edb32b1 nodeName:}" failed. No retries permitted until 2026-09-10 17:41:43.720314881 +0000 UTC m=+1072.429909125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk" (UID: "c0121041-22a8-4c21-8ad7-da8f9edb32b1") : secret "maas-api-e2e-shared-a-w2-c990a5-serving-cert" not found
Sep 10 17:41:42.764502 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.764472    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"]
Sep 10 17:41:42.768673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.768652    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:42.771633 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.771608    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c990a5-dockercfg-dqt5v\""
Sep 10 17:41:42.771834 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.771793    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c990a5\""
Sep 10 17:41:42.778705 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.778681    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"]
Sep 10 17:41:42.805500 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.805476    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"]
Sep 10 17:41:42.808995 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.808975    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:42.820577 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.820554    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"]
Sep 10 17:41:42.921346 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.921312    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw\" (UID: \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:42.921510 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.921370    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f6g2\" (UniqueName: \"kubernetes.io/projected/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-kube-api-access-2f6g2\") pod \"payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw\" (UID: \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:42.921510 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.921417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2sk\" (UniqueName: \"kubernetes.io/projected/1183b65f-498a-462d-9206-22acb94131fa-kube-api-access-7v2sk\") pod \"payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl\" (UID: \"1183b65f-498a-462d-9206-22acb94131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:42.921510 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:42.921472    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1183b65f-498a-462d-9206-22acb94131fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl\" (UID: \"1183b65f-498a-462d-9206-22acb94131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:43.022244 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.022207    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2sk\" (UniqueName: \"kubernetes.io/projected/1183b65f-498a-462d-9206-22acb94131fa-kube-api-access-7v2sk\") pod \"payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl\" (UID: \"1183b65f-498a-462d-9206-22acb94131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:43.022614 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.022279    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1183b65f-498a-462d-9206-22acb94131fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl\" (UID: \"1183b65f-498a-462d-9206-22acb94131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:43.022614 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.022340    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw\" (UID: \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:43.022614 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.022381    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6g2\" (UniqueName: \"kubernetes.io/projected/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-kube-api-access-2f6g2\") pod \"payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw\" (UID: \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:43.023008 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.022987    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1183b65f-498a-462d-9206-22acb94131fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl\" (UID: \"1183b65f-498a-462d-9206-22acb94131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:43.023072 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.023055    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw\" (UID: \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:43.031315 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.031290    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6g2\" (UniqueName: \"kubernetes.io/projected/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-kube-api-access-2f6g2\") pod \"payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw\" (UID: \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:43.031449 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.031398    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2sk\" (UniqueName: \"kubernetes.io/projected/1183b65f-498a-462d-9206-22acb94131fa-kube-api-access-7v2sk\") pod \"payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl\" (UID: \"1183b65f-498a-462d-9206-22acb94131fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:43.080836 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.080805    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:43.119077 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.119043    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:43.353946 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.353924    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"]
Sep 10 17:41:43.355975 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:41:43.355950    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35db41c7_3708_4b9b_b78d_1a24ae1a0d1d.slice/crio-7f56784a42619887b8b68ddf2f43b01e5a58dcab48c2100208789187a3d5774d WatchSource:0}: Error finding container 7f56784a42619887b8b68ddf2f43b01e5a58dcab48c2100208789187a3d5774d: Status 404 returned error can't find the container with id 7f56784a42619887b8b68ddf2f43b01e5a58dcab48c2100208789187a3d5774d
Sep 10 17:41:43.412514 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.412279    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"]
Sep 10 17:41:43.415370 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:41:43.415339    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1183b65f_498a_462d_9206_22acb94131fa.slice/crio-0db5764773c6894cc86e69628e1c46fddcd68b6eb9f2c552206d2c46100c874f WatchSource:0}: Error finding container 0db5764773c6894cc86e69628e1c46fddcd68b6eb9f2c552206d2c46100c874f: Status 404 returned error can't find the container with id 0db5764773c6894cc86e69628e1c46fddcd68b6eb9f2c552206d2c46100c874f
Sep 10 17:41:43.728690 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.728592    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:43.730960 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.730929    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:43.834335 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.834300    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:43.904987 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.904951    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"]
Sep 10 17:41:43.909717 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.909701    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:43.912757 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.912738    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-058069-serving-cert\""
Sep 10 17:41:43.919712 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.919492    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"]
Sep 10 17:41:43.946394 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.941559    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl" event={"ID":"1183b65f-498a-462d-9206-22acb94131fa","Type":"ContainerStarted","Data":"4b886b9b3b9e11968f5008d3de86766e917070c094b6fc1c5aaf8c7c366aa405"}
Sep 10 17:41:43.946394 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.941616    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl" event={"ID":"1183b65f-498a-462d-9206-22acb94131fa","Type":"ContainerStarted","Data":"0db5764773c6894cc86e69628e1c46fddcd68b6eb9f2c552206d2c46100c874f"}
Sep 10 17:41:43.946394 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.942392    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:41:43.946394 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.944300    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw" event={"ID":"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d","Type":"ContainerStarted","Data":"a708568c2565851abf909f9634d81cfe161b300a65434943be432b56831a6b9d"}
Sep 10 17:41:43.946394 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.944356    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw" event={"ID":"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d","Type":"ContainerStarted","Data":"7f56784a42619887b8b68ddf2f43b01e5a58dcab48c2100208789187a3d5774d"}
Sep 10 17:41:43.946394 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.944896    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:43.984222 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:43.984111    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl" podStartSLOduration=1.984093895 podStartE2EDuration="1.984093895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:43.983577623 +0000 UTC m=+1072.693171892" watchObservedRunningTime="2026-09-10 17:41:43.984093895 +0000 UTC m=+1072.693688160"
Sep 10 17:41:44.031627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.031543    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.031627 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.031585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczsb\" (UniqueName: \"kubernetes.io/projected/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-kube-api-access-rczsb\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.032013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.031915    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.032013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.031979    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.113589 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.113360    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw" podStartSLOduration=2.113340124 podStartE2EDuration="2.113340124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:44.012646822 +0000 UTC m=+1072.722241087" watchObservedRunningTime="2026-09-10 17:41:44.113340124 +0000 UTC m=+1072.822934391"
Sep 10 17:41:44.114464 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.114445    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"]
Sep 10 17:41:44.116405 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:41:44.116374    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0121041_22a8_4c21_8ad7_da8f9edb32b1.slice/crio-4d809f7d5da99dc1cfbe355c92280e8fbc5effb4ab821608a1d32fb7ad22fb98 WatchSource:0}: Error finding container 4d809f7d5da99dc1cfbe355c92280e8fbc5effb4ab821608a1d32fb7ad22fb98: Status 404 returned error can't find the container with id 4d809f7d5da99dc1cfbe355c92280e8fbc5effb4ab821608a1d32fb7ad22fb98
Sep 10 17:41:44.132768 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.132733    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.132768 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.132770    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczsb\" (UniqueName: \"kubernetes.io/projected/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-kube-api-access-rczsb\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.132953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.132805    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.132953 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.132838    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.132953 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:44.132900    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-serving-cert: secret "maas-api-e2e-shared-a-w3-058069-serving-cert" not found
Sep 10 17:41:44.133119 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:41:44.132960    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls podName:5e57d2c2-7006-4c61-9bde-acde2cfe1e16 nodeName:}" failed. No retries permitted until 2026-09-10 17:41:44.632940654 +0000 UTC m=+1073.342534912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls") pod "maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2" (UID: "5e57d2c2-7006-4c61-9bde-acde2cfe1e16") : secret "maas-api-e2e-shared-a-w3-058069-serving-cert" not found
Sep 10 17:41:44.133533 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.133512    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.135526 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.135504    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.142599 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.142519    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczsb\" (UniqueName: \"kubernetes.io/projected/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-kube-api-access-rczsb\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.637108 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.637074    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.639993 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.639967    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.822892 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.822860    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:44.952780 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.952738    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk" event={"ID":"c0121041-22a8-4c21-8ad7-da8f9edb32b1","Type":"ContainerStarted","Data":"6a035321514a0763cdab6be9c629fe01bf81a3e735ecbcc8d7101e80be683047"}
Sep 10 17:41:44.952947 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.952807    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk" event={"ID":"c0121041-22a8-4c21-8ad7-da8f9edb32b1","Type":"ContainerStarted","Data":"4d809f7d5da99dc1cfbe355c92280e8fbc5effb4ab821608a1d32fb7ad22fb98"}
Sep 10 17:41:44.952947 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.952839    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:44.972572 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:44.972516    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk" podStartSLOduration=3.63961472 podStartE2EDuration="3.972494897s" totalImagesPullingTime="332.880177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:44.970037899 +0000 UTC m=+1073.679632193" watchObservedRunningTime="2026-09-10 17:41:44.972494897 +0000 UTC m=+1073.682089165"
Sep 10 17:41:45.104830 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:45.104801    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"]
Sep 10 17:41:45.106490 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:41:45.106462    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e57d2c2_7006_4c61_9bde_acde2cfe1e16.slice/crio-f4f0fb89eea4677d693e4018079495df6c33e5ff3fe388ecc3903f5f63d1fc2d WatchSource:0}: Error finding container f4f0fb89eea4677d693e4018079495df6c33e5ff3fe388ecc3903f5f63d1fc2d: Status 404 returned error can't find the container with id f4f0fb89eea4677d693e4018079495df6c33e5ff3fe388ecc3903f5f63d1fc2d
Sep 10 17:41:45.959934 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:45.959897    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2" event={"ID":"5e57d2c2-7006-4c61-9bde-acde2cfe1e16","Type":"ContainerStarted","Data":"f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962"}
Sep 10 17:41:45.960107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:45.959954    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2" event={"ID":"5e57d2c2-7006-4c61-9bde-acde2cfe1e16","Type":"ContainerStarted","Data":"f4f0fb89eea4677d693e4018079495df6c33e5ff3fe388ecc3903f5f63d1fc2d"}
Sep 10 17:41:45.960169 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:45.960115    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:45.979217 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:45.979170    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2" podStartSLOduration=2.628313223 podStartE2EDuration="2.979156162s" totalImagesPullingTime="350.842939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:45.977213959 +0000 UTC m=+1074.686808245" watchObservedRunningTime="2026-09-10 17:41:45.979156162 +0000 UTC m=+1074.688750465"
Sep 10 17:41:50.966056 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:50.966028    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:41:51.970406 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:51.970377    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:41:55.962815 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:55.962783    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:41:55.963316 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:41:55.962861    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:42:10.495845 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.495813    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"]
Sep 10 17:42:10.500333 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.500310    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.503211 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.503187    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-762738-serving-cert\""
Sep 10 17:42:10.512864 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.512843    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"]
Sep 10 17:42:10.559699 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.559663    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.559894 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.559721    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.559894 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.559829    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkkn\" (UniqueName: \"kubernetes.io/projected/7e17b7ce-6430-4afc-9968-06c1f09f55d1-kube-api-access-fjkkn\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.559976 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.559895    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e17b7ce-6430-4afc-9968-06c1f09f55d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.660666 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.660623    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkkn\" (UniqueName: \"kubernetes.io/projected/7e17b7ce-6430-4afc-9968-06c1f09f55d1-kube-api-access-fjkkn\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.660841 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.660702    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e17b7ce-6430-4afc-9968-06c1f09f55d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.660841 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.660740    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.660841 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.660804    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.661033 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:42:10.660911    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-serving-cert: secret "maas-api-e2e-shared-b-w2-762738-serving-cert" not found
Sep 10 17:42:10.661033 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:42:10.660989    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls podName:7e17b7ce-6430-4afc-9968-06c1f09f55d1 nodeName:}" failed. No retries permitted until 2026-09-10 17:42:11.160966025 +0000 UTC m=+1099.870560286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls") pod "maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25" (UID: "7e17b7ce-6430-4afc-9968-06c1f09f55d1") : secret "maas-api-e2e-shared-b-w2-762738-serving-cert" not found
Sep 10 17:42:10.661476 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.661452    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e17b7ce-6430-4afc-9968-06c1f09f55d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.663271 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.663229    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:10.680822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:10.680796    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkkn\" (UniqueName: \"kubernetes.io/projected/7e17b7ce-6430-4afc-9968-06c1f09f55d1-kube-api-access-fjkkn\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:11.165519 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.165476    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:11.167997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.167974    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:11.181167 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.181129    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"]
Sep 10 17:42:11.184874 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.184858    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:11.187530 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.187508    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-762738-dockercfg-942m5\""
Sep 10 17:42:11.187530 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.187522    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-762738\""
Sep 10 17:42:11.195870 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.195849    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"]
Sep 10 17:42:11.266794 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.266748    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/ef3f1703-d25d-4cec-a6a4-824da7a9e221-kube-api-access-w2rf6\") pod \"payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7\" (UID: \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:11.266961 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.266918    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3f1703-d25d-4cec-a6a4-824da7a9e221-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7\" (UID: \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:11.367449 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.367408    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3f1703-d25d-4cec-a6a4-824da7a9e221-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7\" (UID: \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:11.367766 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.367462    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/ef3f1703-d25d-4cec-a6a4-824da7a9e221-kube-api-access-w2rf6\") pod \"payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7\" (UID: \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:11.368165 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.368143    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3f1703-d25d-4cec-a6a4-824da7a9e221-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7\" (UID: \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:11.376888 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.376860    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/ef3f1703-d25d-4cec-a6a4-824da7a9e221-kube-api-access-w2rf6\") pod \"payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7\" (UID: \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:11.413564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.413528    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:11.496569 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.496532    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:11.691930 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.691898    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"]
Sep 10 17:42:11.693937 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:42:11.693914    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e17b7ce_6430_4afc_9968_06c1f09f55d1.slice/crio-d830cff70fac3b56062ef2bfc04040972594463f76bf61111da195f4f9711e4e WatchSource:0}: Error finding container d830cff70fac3b56062ef2bfc04040972594463f76bf61111da195f4f9711e4e: Status 404 returned error can't find the container with id d830cff70fac3b56062ef2bfc04040972594463f76bf61111da195f4f9711e4e
Sep 10 17:42:11.776431 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:11.776409    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"]
Sep 10 17:42:11.778503 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:42:11.778473    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3f1703_d25d_4cec_a6a4_824da7a9e221.slice/crio-f2dd74e9c0ce550cbc57a9d6039551d718eead46caada28313bacfabba1d3984 WatchSource:0}: Error finding container f2dd74e9c0ce550cbc57a9d6039551d718eead46caada28313bacfabba1d3984: Status 404 returned error can't find the container with id f2dd74e9c0ce550cbc57a9d6039551d718eead46caada28313bacfabba1d3984
Sep 10 17:42:12.091833 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.091702    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7" event={"ID":"ef3f1703-d25d-4cec-a6a4-824da7a9e221","Type":"ContainerStarted","Data":"c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688"}
Sep 10 17:42:12.091833 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.091770    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:12.091833 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.091798    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7" event={"ID":"ef3f1703-d25d-4cec-a6a4-824da7a9e221","Type":"ContainerStarted","Data":"f2dd74e9c0ce550cbc57a9d6039551d718eead46caada28313bacfabba1d3984"}
Sep 10 17:42:12.093475 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.093449    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25" event={"ID":"7e17b7ce-6430-4afc-9968-06c1f09f55d1","Type":"ContainerStarted","Data":"4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec"}
Sep 10 17:42:12.093569 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.093495    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25" event={"ID":"7e17b7ce-6430-4afc-9968-06c1f09f55d1","Type":"ContainerStarted","Data":"d830cff70fac3b56062ef2bfc04040972594463f76bf61111da195f4f9711e4e"}
Sep 10 17:42:12.093569 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.093526    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:12.111990 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.111939    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7" podStartSLOduration=1.111922824 podStartE2EDuration="1.111922824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:12.110267192 +0000 UTC m=+1100.819861462" watchObservedRunningTime="2026-09-10 17:42:12.111922824 +0000 UTC m=+1100.821517090"
Sep 10 17:42:12.131510 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.131454    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25" podStartSLOduration=1.830572046 podStartE2EDuration="2.131437773s" totalImagesPullingTime="300.865727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:12.129469576 +0000 UTC m=+1100.839063839" watchObservedRunningTime="2026-09-10 17:42:12.131437773 +0000 UTC m=+1100.841032039"
Sep 10 17:42:12.701675 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.701642    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"]
Sep 10 17:42:12.705440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.705415    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.708498 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.708475    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4f14bf-serving-cert\""
Sep 10 17:42:12.717895 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.717869    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"]
Sep 10 17:42:12.781076 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.781041    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.781310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.781102    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wwnx\" (UniqueName: \"kubernetes.io/projected/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-kube-api-access-9wwnx\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.781310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.781124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.781310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.781173    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.881923 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.881889    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.882100 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.881942    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wwnx\" (UniqueName: \"kubernetes.io/projected/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-kube-api-access-9wwnx\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.882100 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.881960    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.882100 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.881978    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.882100 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:42:12.882061    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-serving-cert: secret "maas-api-e2e-shared-b-w3-4f14bf-serving-cert" not found
Sep 10 17:42:12.882288 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:42:12.882136    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls podName:ccfb3590-2d47-4e09-a2b3-2f34242aaaf8 nodeName:}" failed. No retries permitted until 2026-09-10 17:42:13.382115152 +0000 UTC m=+1102.091709397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls") pod "maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56" (UID: "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8") : secret "maas-api-e2e-shared-b-w3-4f14bf-serving-cert" not found
Sep 10 17:42:12.882592 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.882572    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.884436 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.884416    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:12.890553 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:12.890532    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wwnx\" (UniqueName: \"kubernetes.io/projected/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-kube-api-access-9wwnx\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:13.386987 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:13.386955    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:13.389391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:13.389366    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:13.619192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:13.619160    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:13.896831 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:13.896805    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"]
Sep 10 17:42:13.899080 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:42:13.899052    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfb3590_2d47_4e09_a2b3_2f34242aaaf8.slice/crio-9f6ad73b04c8e3d2b4e62f03338b3214bc412d7adc1669ce8e5d892839a3480e WatchSource:0}: Error finding container 9f6ad73b04c8e3d2b4e62f03338b3214bc412d7adc1669ce8e5d892839a3480e: Status 404 returned error can't find the container with id 9f6ad73b04c8e3d2b4e62f03338b3214bc412d7adc1669ce8e5d892839a3480e
Sep 10 17:42:14.105117 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:14.105081    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56" event={"ID":"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8","Type":"ContainerStarted","Data":"9f6ad73b04c8e3d2b4e62f03338b3214bc412d7adc1669ce8e5d892839a3480e"}
Sep 10 17:42:15.111293 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:15.111256    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56" event={"ID":"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8","Type":"ContainerStarted","Data":"2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796"}
Sep 10 17:42:15.111686 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:15.111378    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:15.129893 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:15.129841    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56" podStartSLOduration=2.832678175 podStartE2EDuration="3.129826101s" totalImagesPullingTime="297.147926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:15.129187767 +0000 UTC m=+1103.838782033" watchObservedRunningTime="2026-09-10 17:42:15.129826101 +0000 UTC m=+1103.839420367"
Sep 10 17:42:18.104914 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:18.104878    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:42:21.123027 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:21.122997    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:42:23.101430 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:23.101399    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:42:50.617365 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.617332    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"]
Sep 10 17:42:50.621123 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.621101    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.623876 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.623842    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1407ad96-serving-cert\""
Sep 10 17:42:50.632251 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.632161    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"]
Sep 10 17:42:50.637105 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.637071    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.637284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.637122    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-metrics-certs\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.637284 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.637242    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhpdm\" (UniqueName: \"kubernetes.io/projected/a3d83ec7-3d3f-4167-a091-08313005b75e-kube-api-access-lhpdm\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.637419 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.637295    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d83ec7-3d3f-4167-a091-08313005b75e-openshift-service-ca\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.738119 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.738081    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.738119 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.738119    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-metrics-certs\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.738412 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.738168    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhpdm\" (UniqueName: \"kubernetes.io/projected/a3d83ec7-3d3f-4167-a091-08313005b75e-kube-api-access-lhpdm\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.738412 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.738203    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d83ec7-3d3f-4167-a091-08313005b75e-openshift-service-ca\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.738412 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:42:50.738222    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-serving-cert: secret "maas-api-e2e-mt-1407ad96-serving-cert" not found
Sep 10 17:42:50.738412 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:42:50.738315    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls podName:a3d83ec7-3d3f-4167-a091-08313005b75e nodeName:}" failed. No retries permitted until 2026-09-10 17:42:51.238298707 +0000 UTC m=+1139.947892950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls") pod "maas-api-e2e-mt-1407ad96-64db7c445c-7klkn" (UID: "a3d83ec7-3d3f-4167-a091-08313005b75e") : secret "maas-api-e2e-mt-1407ad96-serving-cert" not found
Sep 10 17:42:50.738842 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.738823    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d83ec7-3d3f-4167-a091-08313005b75e-openshift-service-ca\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.740837 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.740814    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-metrics-certs\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:50.747664 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:50.747632    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhpdm\" (UniqueName: \"kubernetes.io/projected/a3d83ec7-3d3f-4167-a091-08313005b75e-kube-api-access-lhpdm\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:51.241607 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:51.241564    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:51.244468 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:51.244434    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls\") pod \"maas-api-e2e-mt-1407ad96-64db7c445c-7klkn\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:51.534784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:51.534738    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:51.821928 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:51.821903    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"]
Sep 10 17:42:51.823917 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:42:51.823892    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d83ec7_3d3f_4167_a091_08313005b75e.slice/crio-33185d6e5bdbfede5442051d3cf5ef8058d181c7a4cc73f6c2eaccc91568d77f WatchSource:0}: Error finding container 33185d6e5bdbfede5442051d3cf5ef8058d181c7a4cc73f6c2eaccc91568d77f: Status 404 returned error can't find the container with id 33185d6e5bdbfede5442051d3cf5ef8058d181c7a4cc73f6c2eaccc91568d77f
Sep 10 17:42:52.314073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:52.314037    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn" event={"ID":"a3d83ec7-3d3f-4167-a091-08313005b75e","Type":"ContainerStarted","Data":"33185d6e5bdbfede5442051d3cf5ef8058d181c7a4cc73f6c2eaccc91568d77f"}
Sep 10 17:42:53.320595 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:53.320560    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn" event={"ID":"a3d83ec7-3d3f-4167-a091-08313005b75e","Type":"ContainerStarted","Data":"6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74"}
Sep 10 17:42:53.321003 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:53.320670    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:42:53.342246 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:53.342185    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn" podStartSLOduration=2.917314028 podStartE2EDuration="3.34216296s" totalImagesPullingTime="424.848932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:53.340982234 +0000 UTC m=+1142.050576514" watchObservedRunningTime="2026-09-10 17:42:53.34216296 +0000 UTC m=+1142.051757227"
Sep 10 17:42:59.331195 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:42:59.331159    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:43:51.786310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:43:51.786278    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:43:51.788281 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:43:51.788261    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:44:30.361384 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.361348    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"]
Sep 10 17:44:30.363728 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.361551    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn" podUID="a3d83ec7-3d3f-4167-a091-08313005b75e" containerName="maas-api" containerID="cri-o://6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74" gracePeriod=30
Sep 10 17:44:30.616454 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.616423    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:44:30.654051 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.654023    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d83ec7-3d3f-4167-a091-08313005b75e-openshift-service-ca\") pod \"a3d83ec7-3d3f-4167-a091-08313005b75e\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") "
Sep 10 17:44:30.654270 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.654064    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhpdm\" (UniqueName: \"kubernetes.io/projected/a3d83ec7-3d3f-4167-a091-08313005b75e-kube-api-access-lhpdm\") pod \"a3d83ec7-3d3f-4167-a091-08313005b75e\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") "
Sep 10 17:44:30.654270 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.654099    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls\") pod \"a3d83ec7-3d3f-4167-a091-08313005b75e\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") "
Sep 10 17:44:30.654270 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.654168    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-metrics-certs\") pod \"a3d83ec7-3d3f-4167-a091-08313005b75e\" (UID: \"a3d83ec7-3d3f-4167-a091-08313005b75e\") "
Sep 10 17:44:30.655837 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.654494    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d83ec7-3d3f-4167-a091-08313005b75e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3d83ec7-3d3f-4167-a091-08313005b75e" (UID: "a3d83ec7-3d3f-4167-a091-08313005b75e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:44:30.655837 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.654641    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d83ec7-3d3f-4167-a091-08313005b75e-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:44:30.657214 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.657140    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d83ec7-3d3f-4167-a091-08313005b75e-kube-api-access-lhpdm" (OuterVolumeSpecName: "kube-api-access-lhpdm") pod "a3d83ec7-3d3f-4167-a091-08313005b75e" (UID: "a3d83ec7-3d3f-4167-a091-08313005b75e"). InnerVolumeSpecName "kube-api-access-lhpdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:44:30.657344 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.657275    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3d83ec7-3d3f-4167-a091-08313005b75e" (UID: "a3d83ec7-3d3f-4167-a091-08313005b75e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:44:30.658711 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.658688    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a3d83ec7-3d3f-4167-a091-08313005b75e" (UID: "a3d83ec7-3d3f-4167-a091-08313005b75e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:44:30.756069 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.755996    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:44:30.756069 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.756024    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhpdm\" (UniqueName: \"kubernetes.io/projected/a3d83ec7-3d3f-4167-a091-08313005b75e-kube-api-access-lhpdm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:44:30.756069 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.756037    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3d83ec7-3d3f-4167-a091-08313005b75e-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:44:30.835913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.835878    2568 generic.go:356] "Generic (PLEG): container finished" podID="a3d83ec7-3d3f-4167-a091-08313005b75e" containerID="6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74" exitCode=0
Sep 10 17:44:30.836078 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.835926    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"
Sep 10 17:44:30.836078 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.835963    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn" event={"ID":"a3d83ec7-3d3f-4167-a091-08313005b75e","Type":"ContainerDied","Data":"6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74"}
Sep 10 17:44:30.836078 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.836022    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn" event={"ID":"a3d83ec7-3d3f-4167-a091-08313005b75e","Type":"ContainerDied","Data":"33185d6e5bdbfede5442051d3cf5ef8058d181c7a4cc73f6c2eaccc91568d77f"}
Sep 10 17:44:30.836078 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.836052    2568 scope.go:117] "RemoveContainer" containerID="6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74"
Sep 10 17:44:30.846928 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.846906    2568 scope.go:117] "RemoveContainer" containerID="6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74"
Sep 10 17:44:30.847284 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:44:30.847254    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74\": container with ID starting with 6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74 not found: ID does not exist" containerID="6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74"
Sep 10 17:44:30.847406 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.847291    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74"} err="failed to get container status \"6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74\": rpc error: code = NotFound desc = could not find container \"6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74\": container with ID starting with 6e116e8beec8cbc699bff01834aabd908f8b137826f9ef5523efa1062add2a74 not found: ID does not exist"
Sep 10 17:44:30.860198 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.860170    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"]
Sep 10 17:44:30.866625 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:30.866602    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1407ad96-64db7c445c-7klkn"]
Sep 10 17:44:31.765108 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:44:31.765069    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d83ec7-3d3f-4167-a091-08313005b75e" path="/var/lib/kubelet/pods/a3d83ec7-3d3f-4167-a091-08313005b75e/volumes"
Sep 10 17:45:07.717317 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.717277    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7764-f3f90f"]
Sep 10 17:45:07.717797 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.717693    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d83ec7-3d3f-4167-a091-08313005b75e" containerName="maas-api"
Sep 10 17:45:07.717797 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.717704    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d83ec7-3d3f-4167-a091-08313005b75e" containerName="maas-api"
Sep 10 17:45:07.717797 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.717763    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d83ec7-3d3f-4167-a091-08313005b75e" containerName="maas-api"
Sep 10 17:45:07.721985 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.721968    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-f3f90f"
Sep 10 17:45:07.724766 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.724742    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p5m84\""
Sep 10 17:45:07.728367 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.728341    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-f3f90f"]
Sep 10 17:45:07.803475 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.803444    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrxgb\" (UniqueName: \"kubernetes.io/projected/7c1d12bf-0580-4937-8b61-1a8740d4e38a-kube-api-access-nrxgb\") pod \"test-curl-7764-f3f90f\" (UID: \"7c1d12bf-0580-4937-8b61-1a8740d4e38a\") " pod="opendatahub/test-curl-7764-f3f90f"
Sep 10 17:45:07.904081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.904038    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrxgb\" (UniqueName: \"kubernetes.io/projected/7c1d12bf-0580-4937-8b61-1a8740d4e38a-kube-api-access-nrxgb\") pod \"test-curl-7764-f3f90f\" (UID: \"7c1d12bf-0580-4937-8b61-1a8740d4e38a\") " pod="opendatahub/test-curl-7764-f3f90f"
Sep 10 17:45:07.913411 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:07.913384    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrxgb\" (UniqueName: \"kubernetes.io/projected/7c1d12bf-0580-4937-8b61-1a8740d4e38a-kube-api-access-nrxgb\") pod \"test-curl-7764-f3f90f\" (UID: \"7c1d12bf-0580-4937-8b61-1a8740d4e38a\") " pod="opendatahub/test-curl-7764-f3f90f"
Sep 10 17:45:08.033391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:08.033364    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-f3f90f"
Sep 10 17:45:08.303101 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:08.303037    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-f3f90f"]
Sep 10 17:45:08.305544 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:45:08.305515    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c1d12bf_0580_4937_8b61_1a8740d4e38a.slice/crio-962f1492fb16724e359398f8dc7037be6dc9e7e5a96d990da3da5170e55d9b1e WatchSource:0}: Error finding container 962f1492fb16724e359398f8dc7037be6dc9e7e5a96d990da3da5170e55d9b1e: Status 404 returned error can't find the container with id 962f1492fb16724e359398f8dc7037be6dc9e7e5a96d990da3da5170e55d9b1e
Sep 10 17:45:09.033721 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:09.033687    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-f3f90f" event={"ID":"7c1d12bf-0580-4937-8b61-1a8740d4e38a","Type":"ContainerStarted","Data":"e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c"}
Sep 10 17:45:09.034107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:09.033735    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-f3f90f" event={"ID":"7c1d12bf-0580-4937-8b61-1a8740d4e38a","Type":"ContainerStarted","Data":"962f1492fb16724e359398f8dc7037be6dc9e7e5a96d990da3da5170e55d9b1e"}
Sep 10 17:45:09.052392 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:09.052342    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7764-f3f90f" podStartSLOduration=2.052324198 podStartE2EDuration="2.052324198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:09.051450555 +0000 UTC m=+1277.761044822" watchObservedRunningTime="2026-09-10 17:45:09.052324198 +0000 UTC m=+1277.761918463"
Sep 10 17:45:09.654147 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:09.654109    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7764-f3f90f"]
Sep 10 17:45:09.658867 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:09.658837    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7764-f3f90f"]
Sep 10 17:45:10.038478 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:45:10.038450    2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7764-f3f90f" container="test-curl-7764-f3f90f"
Sep 10 17:45:11.043667 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:11.043377    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7764-f3f90f" podUID="7c1d12bf-0580-4937-8b61-1a8740d4e38a" containerName="test-curl-7764-f3f90f" containerID="cri-o://e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c" gracePeriod=2
Sep 10 17:45:13.240292 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:13.240230    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-f3f90f"
Sep 10 17:45:13.250343 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:13.250320    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrxgb\" (UniqueName: \"kubernetes.io/projected/7c1d12bf-0580-4937-8b61-1a8740d4e38a-kube-api-access-nrxgb\") pod \"7c1d12bf-0580-4937-8b61-1a8740d4e38a\" (UID: \"7c1d12bf-0580-4937-8b61-1a8740d4e38a\") "
Sep 10 17:45:13.252320 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:13.252294    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1d12bf-0580-4937-8b61-1a8740d4e38a-kube-api-access-nrxgb" (OuterVolumeSpecName: "kube-api-access-nrxgb") pod "7c1d12bf-0580-4937-8b61-1a8740d4e38a" (UID: "7c1d12bf-0580-4937-8b61-1a8740d4e38a"). InnerVolumeSpecName "kube-api-access-nrxgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:13.351984 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:13.351893    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrxgb\" (UniqueName: \"kubernetes.io/projected/7c1d12bf-0580-4937-8b61-1a8740d4e38a-kube-api-access-nrxgb\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:45:13.772452 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:13.772412    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1d12bf-0580-4937-8b61-1a8740d4e38a" path="/var/lib/kubelet/pods/7c1d12bf-0580-4937-8b61-1a8740d4e38a/volumes"
Sep 10 17:45:14.061065 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:14.060972    2568 generic.go:356] "Generic (PLEG): container finished" podID="7c1d12bf-0580-4937-8b61-1a8740d4e38a" containerID="e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c" exitCode=137
Sep 10 17:45:14.061065 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:14.061034    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-f3f90f"
Sep 10 17:45:14.061297 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:14.061077    2568 scope.go:117] "RemoveContainer" containerID="e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c"
Sep 10 17:45:14.069772 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:14.069750    2568 scope.go:117] "RemoveContainer" containerID="e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c"
Sep 10 17:45:14.070059 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:45:14.070038    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c\": container with ID starting with e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c not found: ID does not exist" containerID="e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c"
Sep 10 17:45:14.070125 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:45:14.070068    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c"} err="failed to get container status \"e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c\": rpc error: code = NotFound desc = could not find container \"e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c\": container with ID starting with e8ad008b29b57e379fd4d9e969f6b14b815c1715e979eba103425123c922588c not found: ID does not exist"
Sep 10 17:46:34.605946 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.605902    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7764-146760"]
Sep 10 17:46:34.606557 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.606537    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1d12bf-0580-4937-8b61-1a8740d4e38a" containerName="test-curl-7764-f3f90f"
Sep 10 17:46:34.606643 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.606559    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1d12bf-0580-4937-8b61-1a8740d4e38a" containerName="test-curl-7764-f3f90f"
Sep 10 17:46:34.606701 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.606658    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1d12bf-0580-4937-8b61-1a8740d4e38a" containerName="test-curl-7764-f3f90f"
Sep 10 17:46:34.609881 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.609862    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-146760"
Sep 10 17:46:34.612547 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.612527    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p5m84\""
Sep 10 17:46:34.616300 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.616274    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-146760"]
Sep 10 17:46:34.710724 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.710668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2bz\" (UniqueName: \"kubernetes.io/projected/fcb07bb7-524c-4234-930f-59ed9d13ba72-kube-api-access-fb2bz\") pod \"test-curl-7764-146760\" (UID: \"fcb07bb7-524c-4234-930f-59ed9d13ba72\") " pod="opendatahub/test-curl-7764-146760"
Sep 10 17:46:34.812182 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.812147    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2bz\" (UniqueName: \"kubernetes.io/projected/fcb07bb7-524c-4234-930f-59ed9d13ba72-kube-api-access-fb2bz\") pod \"test-curl-7764-146760\" (UID: \"fcb07bb7-524c-4234-930f-59ed9d13ba72\") " pod="opendatahub/test-curl-7764-146760"
Sep 10 17:46:34.820579 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.820547    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2bz\" (UniqueName: \"kubernetes.io/projected/fcb07bb7-524c-4234-930f-59ed9d13ba72-kube-api-access-fb2bz\") pod \"test-curl-7764-146760\" (UID: \"fcb07bb7-524c-4234-930f-59ed9d13ba72\") " pod="opendatahub/test-curl-7764-146760"
Sep 10 17:46:34.920548 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:34.920474    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-146760"
Sep 10 17:46:35.190833 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:35.190806    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7764-146760"]
Sep 10 17:46:35.193143 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:46:35.193112    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcb07bb7_524c_4234_930f_59ed9d13ba72.slice/crio-267068c484bb16e641af2552a0e312e823e69eb5781a420d7b302a22e3e5a6b6 WatchSource:0}: Error finding container 267068c484bb16e641af2552a0e312e823e69eb5781a420d7b302a22e3e5a6b6: Status 404 returned error can't find the container with id 267068c484bb16e641af2552a0e312e823e69eb5781a420d7b302a22e3e5a6b6
Sep 10 17:46:35.480921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:35.480835    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-146760" event={"ID":"fcb07bb7-524c-4234-930f-59ed9d13ba72","Type":"ContainerStarted","Data":"2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56"}
Sep 10 17:46:35.480921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:35.480875    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7764-146760" event={"ID":"fcb07bb7-524c-4234-930f-59ed9d13ba72","Type":"ContainerStarted","Data":"267068c484bb16e641af2552a0e312e823e69eb5781a420d7b302a22e3e5a6b6"}
Sep 10 17:46:35.499271 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:35.499208    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7764-146760" podStartSLOduration=1.499192869 podStartE2EDuration="1.499192869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:46:35.497130064 +0000 UTC m=+1364.206724330" watchObservedRunningTime="2026-09-10 17:46:35.499192869 +0000 UTC m=+1364.208787136"
Sep 10 17:46:36.114378 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:36.114343    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7764-146760"]
Sep 10 17:46:36.119956 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:36.119875    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7764-146760"]
Sep 10 17:46:36.486407 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:46:36.486377    2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7764-146760" container="test-curl-7764-146760"
Sep 10 17:46:37.490975 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:37.490936    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7764-146760" podUID="fcb07bb7-524c-4234-930f-59ed9d13ba72" containerName="test-curl-7764-146760" containerID="cri-o://2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56" gracePeriod=2
Sep 10 17:46:37.493258 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:46:37.493221    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7764-146760\" is forbidden: User \"system:node:ip-10-0-142-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-242.ec2.internal' and this object" podUID="fcb07bb7-524c-4234-930f-59ed9d13ba72" pod="opendatahub/test-curl-7764-146760"
Sep 10 17:46:39.643501 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:39.643468    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-146760"
Sep 10 17:46:39.657851 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:39.657823    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb2bz\" (UniqueName: \"kubernetes.io/projected/fcb07bb7-524c-4234-930f-59ed9d13ba72-kube-api-access-fb2bz\") pod \"fcb07bb7-524c-4234-930f-59ed9d13ba72\" (UID: \"fcb07bb7-524c-4234-930f-59ed9d13ba72\") "
Sep 10 17:46:39.659816 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:39.659793    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb07bb7-524c-4234-930f-59ed9d13ba72-kube-api-access-fb2bz" (OuterVolumeSpecName: "kube-api-access-fb2bz") pod "fcb07bb7-524c-4234-930f-59ed9d13ba72" (UID: "fcb07bb7-524c-4234-930f-59ed9d13ba72"). InnerVolumeSpecName "kube-api-access-fb2bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:46:39.758696 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:39.758606    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb2bz\" (UniqueName: \"kubernetes.io/projected/fcb07bb7-524c-4234-930f-59ed9d13ba72-kube-api-access-fb2bz\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:46:39.764343 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:39.764313    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb07bb7-524c-4234-930f-59ed9d13ba72" path="/var/lib/kubelet/pods/fcb07bb7-524c-4234-930f-59ed9d13ba72/volumes"
Sep 10 17:46:40.509176 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:40.509140    2568 generic.go:356] "Generic (PLEG): container finished" podID="fcb07bb7-524c-4234-930f-59ed9d13ba72" containerID="2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56" exitCode=137
Sep 10 17:46:40.509390 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:40.509190    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7764-146760"
Sep 10 17:46:40.509390 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:40.509201    2568 scope.go:117] "RemoveContainer" containerID="2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56"
Sep 10 17:46:40.517790 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:40.517780    2568 scope.go:117] "RemoveContainer" containerID="2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56"
Sep 10 17:46:40.518067 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:46:40.518046    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56\": container with ID starting with 2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56 not found: ID does not exist" containerID="2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56"
Sep 10 17:46:40.518143 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:40.518077    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56"} err="failed to get container status \"2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56\": rpc error: code = NotFound desc = could not find container \"2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56\": container with ID starting with 2fe94d5b1b041adb6518544024bedcf0accaaaa5e19eb95f459924acac074c56 not found: ID does not exist"
Sep 10 17:46:44.131322 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.131279    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"]
Sep 10 17:46:44.131894 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.131874    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb07bb7-524c-4234-930f-59ed9d13ba72" containerName="test-curl-7764-146760"
Sep 10 17:46:44.131983 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.131897    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb07bb7-524c-4234-930f-59ed9d13ba72" containerName="test-curl-7764-146760"
Sep 10 17:46:44.132037 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.132009    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb07bb7-524c-4234-930f-59ed9d13ba72" containerName="test-curl-7764-146760"
Sep 10 17:46:44.136209 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.136184    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.139166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.139132    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b32e7b-serving-cert\""
Sep 10 17:46:44.146857 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.146830    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"]
Sep 10 17:46:44.200103 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.200069    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tqrl\" (UniqueName: \"kubernetes.io/projected/2241f006-2868-4f0a-ae24-7c8490b90ec4-kube-api-access-8tqrl\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.200103 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.200107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.200340 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.200264    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-metrics-certs\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.200340 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.200295    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2241f006-2868-4f0a-ae24-7c8490b90ec4-openshift-service-ca\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.300976 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.300936    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tqrl\" (UniqueName: \"kubernetes.io/projected/2241f006-2868-4f0a-ae24-7c8490b90ec4-kube-api-access-8tqrl\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.300976 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.300978    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.301200 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.301052    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-metrics-certs\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.301200 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.301069    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2241f006-2868-4f0a-ae24-7c8490b90ec4-openshift-service-ca\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.301200 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:46:44.301152    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-serving-cert: secret "maas-api-e2e-api-a-b32e7b-serving-cert" not found
Sep 10 17:46:44.301372 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:46:44.301215    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls podName:2241f006-2868-4f0a-ae24-7c8490b90ec4 nodeName:}" failed. No retries permitted until 2026-09-10 17:46:44.801198495 +0000 UTC m=+1373.510792739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls") pod "maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z" (UID: "2241f006-2868-4f0a-ae24-7c8490b90ec4") : secret "maas-api-e2e-api-a-b32e7b-serving-cert" not found
Sep 10 17:46:44.301852 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.301834    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2241f006-2868-4f0a-ae24-7c8490b90ec4-openshift-service-ca\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.303657 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.303632    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-metrics-certs\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.311737 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.311718    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tqrl\" (UniqueName: \"kubernetes.io/projected/2241f006-2868-4f0a-ae24-7c8490b90ec4-kube-api-access-8tqrl\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.807170 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:44.807131    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:44.807450 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:46:44.807337    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-serving-cert: secret "maas-api-e2e-api-a-b32e7b-serving-cert" not found
Sep 10 17:46:44.807450 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:46:44.807413    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls podName:2241f006-2868-4f0a-ae24-7c8490b90ec4 nodeName:}" failed. No retries permitted until 2026-09-10 17:46:45.807393649 +0000 UTC m=+1374.516987896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls") pod "maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z" (UID: "2241f006-2868-4f0a-ae24-7c8490b90ec4") : secret "maas-api-e2e-api-a-b32e7b-serving-cert" not found
Sep 10 17:46:45.816169 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:45.816114    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:45.818512 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:45.818492    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls\") pod \"maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:45.949054 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:45.949011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:46.222256 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:46.222208    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"]
Sep 10 17:46:46.224363 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:46:46.224331    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2241f006_2868_4f0a_ae24_7c8490b90ec4.slice/crio-dc793754e38bcd6bc79d3c34adccf8e1f593bad93a3c304e609b90734c753e7b WatchSource:0}: Error finding container dc793754e38bcd6bc79d3c34adccf8e1f593bad93a3c304e609b90734c753e7b: Status 404 returned error can't find the container with id dc793754e38bcd6bc79d3c34adccf8e1f593bad93a3c304e609b90734c753e7b
Sep 10 17:46:46.225764 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:46.225746    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:46:46.543595 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:46.543557    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z" event={"ID":"2241f006-2868-4f0a-ae24-7c8490b90ec4","Type":"ContainerStarted","Data":"dc793754e38bcd6bc79d3c34adccf8e1f593bad93a3c304e609b90734c753e7b"}
Sep 10 17:46:47.551594 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:47.551558    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z" event={"ID":"2241f006-2868-4f0a-ae24-7c8490b90ec4","Type":"ContainerStarted","Data":"cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8"}
Sep 10 17:46:47.551974 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:47.551663    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:46:47.571828 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:47.571773    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z" podStartSLOduration=3.068400265 podStartE2EDuration="3.571758174s" totalImagesPullingTime="503.357909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:46:47.568970424 +0000 UTC m=+1376.278564699" watchObservedRunningTime="2026-09-10 17:46:47.571758174 +0000 UTC m=+1376.281352454"
Sep 10 17:46:53.561594 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:46:53.561564    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:47:13.235378 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.235349    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"]
Sep 10 17:47:13.240156 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.240135    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.243993 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.243971    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-912593-serving-cert\""
Sep 10 17:47:13.252871 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.252836    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"]
Sep 10 17:47:13.375150 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.375114    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-metrics-certs\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.375356 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.375221    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2de142-f1e4-40fc-b076-d6a2f948c82c-openshift-service-ca\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.375356 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.375293    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7vk\" (UniqueName: \"kubernetes.io/projected/1f2de142-f1e4-40fc-b076-d6a2f948c82c-kube-api-access-rp7vk\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.375443 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.375376    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-maas-api-tls\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.476480 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.476434    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-maas-api-tls\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.476480 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.476490    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-metrics-certs\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.476706 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.476548    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2de142-f1e4-40fc-b076-d6a2f948c82c-openshift-service-ca\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.476706 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.476564    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7vk\" (UniqueName: \"kubernetes.io/projected/1f2de142-f1e4-40fc-b076-d6a2f948c82c-kube-api-access-rp7vk\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.477283 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.477228    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2de142-f1e4-40fc-b076-d6a2f948c82c-openshift-service-ca\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.479074 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.479045    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-maas-api-tls\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.479164 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.479106    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-metrics-certs\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.486426 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.486368    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7vk\" (UniqueName: \"kubernetes.io/projected/1f2de142-f1e4-40fc-b076-d6a2f948c82c-kube-api-access-rp7vk\") pod \"maas-api-e2e-api-b-912593-f5c748bc8-997fg\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.554113 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.554078    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:13.836709 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:13.836682    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"]
Sep 10 17:47:13.838682 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:47:13.838645    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2de142_f1e4_40fc_b076_d6a2f948c82c.slice/crio-2fb455e9d4fc100c23d1b474cf476727132a4e53bd7e722a983ba787145485db WatchSource:0}: Error finding container 2fb455e9d4fc100c23d1b474cf476727132a4e53bd7e722a983ba787145485db: Status 404 returned error can't find the container with id 2fb455e9d4fc100c23d1b474cf476727132a4e53bd7e722a983ba787145485db
Sep 10 17:47:14.007560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.007526    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"]
Sep 10 17:47:14.012461 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.012438    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.015583 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.015566    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-912593\""
Sep 10 17:47:14.015666 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.015638    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-912593-dockercfg-tx5nw\""
Sep 10 17:47:14.027199 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.027172    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"]
Sep 10 17:47:14.183300 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.183272    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q4h7\" (UniqueName: \"kubernetes.io/projected/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-kube-api-access-6q4h7\") pod \"payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg\" (UID: \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.183430 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.183383    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg\" (UID: \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.284634 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.284600    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q4h7\" (UniqueName: \"kubernetes.io/projected/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-kube-api-access-6q4h7\") pod \"payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg\" (UID: \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.285105 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.284661    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg\" (UID: \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.285362 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.285343    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg\" (UID: \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.294246 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.294219    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q4h7\" (UniqueName: \"kubernetes.io/projected/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-kube-api-access-6q4h7\") pod \"payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg\" (UID: \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.326268 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.326217    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.604452 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.604420    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"]
Sep 10 17:47:14.606430 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:47:14.606401    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4faa207_7ab6_4f7a_a328_ef64d75b8eb1.slice/crio-24500e9fd83fc33b279f060e2d3da8a5239422da53f012db9fed1650792deac2 WatchSource:0}: Error finding container 24500e9fd83fc33b279f060e2d3da8a5239422da53f012db9fed1650792deac2: Status 404 returned error can't find the container with id 24500e9fd83fc33b279f060e2d3da8a5239422da53f012db9fed1650792deac2
Sep 10 17:47:14.697734 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.697697    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg" event={"ID":"1f2de142-f1e4-40fc-b076-d6a2f948c82c","Type":"ContainerStarted","Data":"8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2"}
Sep 10 17:47:14.697912 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.697762    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg" event={"ID":"1f2de142-f1e4-40fc-b076-d6a2f948c82c","Type":"ContainerStarted","Data":"2fb455e9d4fc100c23d1b474cf476727132a4e53bd7e722a983ba787145485db"}
Sep 10 17:47:14.697912 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.697867    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:14.699360 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.699335    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg" event={"ID":"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1","Type":"ContainerStarted","Data":"563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027"}
Sep 10 17:47:14.699501 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.699384    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg" event={"ID":"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1","Type":"ContainerStarted","Data":"24500e9fd83fc33b279f060e2d3da8a5239422da53f012db9fed1650792deac2"}
Sep 10 17:47:14.699501 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.699409    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:14.725635 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.725592    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg" podStartSLOduration=1.41521979 podStartE2EDuration="1.725575533s" totalImagesPullingTime="310.355743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:14.723605054 +0000 UTC m=+1403.433199322" watchObservedRunningTime="2026-09-10 17:47:14.725575533 +0000 UTC m=+1403.435169798"
Sep 10 17:47:14.748379 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:14.748333    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg" podStartSLOduration=1.74831658 podStartE2EDuration="1.74831658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:14.747345494 +0000 UTC m=+1403.456939761" watchObservedRunningTime="2026-09-10 17:47:14.74831658 +0000 UTC m=+1403.457910846"
Sep 10 17:47:20.710450 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:20.710421    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:47:25.708052 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:25.708022    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:47:55.662633 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.662539    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"]
Sep 10 17:47:55.663276 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.662802    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z" podUID="2241f006-2868-4f0a-ae24-7c8490b90ec4" containerName="maas-api" containerID="cri-o://cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8" gracePeriod=30
Sep 10 17:47:55.910016 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.909989    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:47:55.931840 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.931739    2568 generic.go:356] "Generic (PLEG): container finished" podID="2241f006-2868-4f0a-ae24-7c8490b90ec4" containerID="cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8" exitCode=0
Sep 10 17:47:55.931997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.931846    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"
Sep 10 17:47:55.931997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.931881    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z" event={"ID":"2241f006-2868-4f0a-ae24-7c8490b90ec4","Type":"ContainerDied","Data":"cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8"}
Sep 10 17:47:55.931997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.931933    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z" event={"ID":"2241f006-2868-4f0a-ae24-7c8490b90ec4","Type":"ContainerDied","Data":"dc793754e38bcd6bc79d3c34adccf8e1f593bad93a3c304e609b90734c753e7b"}
Sep 10 17:47:55.931997 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.931966    2568 scope.go:117] "RemoveContainer" containerID="cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8"
Sep 10 17:47:55.943412 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.943382    2568 scope.go:117] "RemoveContainer" containerID="cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8"
Sep 10 17:47:55.943770 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:47:55.943741    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8\": container with ID starting with cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8 not found: ID does not exist" containerID="cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8"
Sep 10 17:47:55.943866 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.943783    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8"} err="failed to get container status \"cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8\": rpc error: code = NotFound desc = could not find container \"cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8\": container with ID starting with cd7fa5b5032a60e1c003b7a0db862844995fa9a6a7eb4abe04e0b35d5ede8fd8 not found: ID does not exist"
Sep 10 17:47:55.973424 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.973398    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls\") pod \"2241f006-2868-4f0a-ae24-7c8490b90ec4\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") "
Sep 10 17:47:55.973598 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.973430    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-metrics-certs\") pod \"2241f006-2868-4f0a-ae24-7c8490b90ec4\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") "
Sep 10 17:47:55.973598 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.973516    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2241f006-2868-4f0a-ae24-7c8490b90ec4-openshift-service-ca\") pod \"2241f006-2868-4f0a-ae24-7c8490b90ec4\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") "
Sep 10 17:47:55.973598 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.973542    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tqrl\" (UniqueName: \"kubernetes.io/projected/2241f006-2868-4f0a-ae24-7c8490b90ec4-kube-api-access-8tqrl\") pod \"2241f006-2868-4f0a-ae24-7c8490b90ec4\" (UID: \"2241f006-2868-4f0a-ae24-7c8490b90ec4\") "
Sep 10 17:47:55.973869 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.973847    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2241f006-2868-4f0a-ae24-7c8490b90ec4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2241f006-2868-4f0a-ae24-7c8490b90ec4" (UID: "2241f006-2868-4f0a-ae24-7c8490b90ec4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:47:55.975639 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.975609    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2241f006-2868-4f0a-ae24-7c8490b90ec4-kube-api-access-8tqrl" (OuterVolumeSpecName: "kube-api-access-8tqrl") pod "2241f006-2868-4f0a-ae24-7c8490b90ec4" (UID: "2241f006-2868-4f0a-ae24-7c8490b90ec4"). InnerVolumeSpecName "kube-api-access-8tqrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:47:55.975639 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.975630    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2241f006-2868-4f0a-ae24-7c8490b90ec4" (UID: "2241f006-2868-4f0a-ae24-7c8490b90ec4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:47:55.975773 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:55.975640    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2241f006-2868-4f0a-ae24-7c8490b90ec4" (UID: "2241f006-2868-4f0a-ae24-7c8490b90ec4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:47:56.074954 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:56.074916    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:47:56.074954 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:56.074946    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2241f006-2868-4f0a-ae24-7c8490b90ec4-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:47:56.074954 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:56.074955    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2241f006-2868-4f0a-ae24-7c8490b90ec4-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:47:56.075197 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:56.074965    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tqrl\" (UniqueName: \"kubernetes.io/projected/2241f006-2868-4f0a-ae24-7c8490b90ec4-kube-api-access-8tqrl\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:47:56.255376 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:56.255340    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"]
Sep 10 17:47:56.259822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:56.259794    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32e7b-657d696c9-5jm7z"]
Sep 10 17:47:57.764536 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:47:57.764499    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2241f006-2868-4f0a-ae24-7c8490b90ec4" path="/var/lib/kubelet/pods/2241f006-2868-4f0a-ae24-7c8490b90ec4/volumes"
Sep 10 17:48:18.171931 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.171876    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"]
Sep 10 17:48:18.172531 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.172388    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg" podUID="1f2de142-f1e4-40fc-b076-d6a2f948c82c" containerName="maas-api" containerID="cri-o://8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2" gracePeriod=30
Sep 10 17:48:18.249014 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.248979    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"]
Sep 10 17:48:18.249313 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.249251    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg" podUID="a4faa207-7ab6-4f7a-a328-ef64d75b8eb1" containerName="payload-pre-processing" containerID="cri-o://563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027" gracePeriod=30
Sep 10 17:48:18.505973 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.505944    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:48:18.547535 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.547507    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:48:18.576439 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.576401    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-maas-api-tls\") pod \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") "
Sep 10 17:48:18.576439 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.576441    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-plugins-config-volume\") pod \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\" (UID: \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\") "
Sep 10 17:48:18.576673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.576471    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2de142-f1e4-40fc-b076-d6a2f948c82c-openshift-service-ca\") pod \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") "
Sep 10 17:48:18.576673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.576520    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp7vk\" (UniqueName: \"kubernetes.io/projected/1f2de142-f1e4-40fc-b076-d6a2f948c82c-kube-api-access-rp7vk\") pod \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") "
Sep 10 17:48:18.576673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.576550    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q4h7\" (UniqueName: \"kubernetes.io/projected/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-kube-api-access-6q4h7\") pod \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\" (UID: \"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1\") "
Sep 10 17:48:18.576673 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.576598    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-metrics-certs\") pod \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\" (UID: \"1f2de142-f1e4-40fc-b076-d6a2f948c82c\") "
Sep 10 17:48:18.576921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.576893    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2de142-f1e4-40fc-b076-d6a2f948c82c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f2de142-f1e4-40fc-b076-d6a2f948c82c" (UID: "1f2de142-f1e4-40fc-b076-d6a2f948c82c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:18.576994 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.576937    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4faa207-7ab6-4f7a-a328-ef64d75b8eb1" (UID: "a4faa207-7ab6-4f7a-a328-ef64d75b8eb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:18.578993 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.578965    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f2de142-f1e4-40fc-b076-d6a2f948c82c" (UID: "1f2de142-f1e4-40fc-b076-d6a2f948c82c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:48:18.579317 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.579275    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-kube-api-access-6q4h7" (OuterVolumeSpecName: "kube-api-access-6q4h7") pod "a4faa207-7ab6-4f7a-a328-ef64d75b8eb1" (UID: "a4faa207-7ab6-4f7a-a328-ef64d75b8eb1"). InnerVolumeSpecName "kube-api-access-6q4h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:18.579397 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.579313    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2de142-f1e4-40fc-b076-d6a2f948c82c-kube-api-access-rp7vk" (OuterVolumeSpecName: "kube-api-access-rp7vk") pod "1f2de142-f1e4-40fc-b076-d6a2f948c82c" (UID: "1f2de142-f1e4-40fc-b076-d6a2f948c82c"). InnerVolumeSpecName "kube-api-access-rp7vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:18.579607 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.579588    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f2de142-f1e4-40fc-b076-d6a2f948c82c" (UID: "1f2de142-f1e4-40fc-b076-d6a2f948c82c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:48:18.677402 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.677357    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:18.677402 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.677397    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2de142-f1e4-40fc-b076-d6a2f948c82c-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:18.677402 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.677407    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp7vk\" (UniqueName: \"kubernetes.io/projected/1f2de142-f1e4-40fc-b076-d6a2f948c82c-kube-api-access-rp7vk\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:18.677402 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.677417    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q4h7\" (UniqueName: \"kubernetes.io/projected/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1-kube-api-access-6q4h7\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:18.677676 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.677427    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:18.677676 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:18.677435    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f2de142-f1e4-40fc-b076-d6a2f948c82c-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:19.053910 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.053875    2568 generic.go:356] "Generic (PLEG): container finished" podID="1f2de142-f1e4-40fc-b076-d6a2f948c82c" containerID="8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2" exitCode=0
Sep 10 17:48:19.054089 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.053921    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"
Sep 10 17:48:19.054089 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.053956    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg" event={"ID":"1f2de142-f1e4-40fc-b076-d6a2f948c82c","Type":"ContainerDied","Data":"8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2"}
Sep 10 17:48:19.054089 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.054012    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg" event={"ID":"1f2de142-f1e4-40fc-b076-d6a2f948c82c","Type":"ContainerDied","Data":"2fb455e9d4fc100c23d1b474cf476727132a4e53bd7e722a983ba787145485db"}
Sep 10 17:48:19.054089 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.054046    2568 scope.go:117] "RemoveContainer" containerID="8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2"
Sep 10 17:48:19.055400 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.055376    2568 generic.go:356] "Generic (PLEG): container finished" podID="a4faa207-7ab6-4f7a-a328-ef64d75b8eb1" containerID="563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027" exitCode=0
Sep 10 17:48:19.055503 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.055459    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"
Sep 10 17:48:19.055503 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.055467    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg" event={"ID":"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1","Type":"ContainerDied","Data":"563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027"}
Sep 10 17:48:19.055624 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.055526    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg" event={"ID":"a4faa207-7ab6-4f7a-a328-ef64d75b8eb1","Type":"ContainerDied","Data":"24500e9fd83fc33b279f060e2d3da8a5239422da53f012db9fed1650792deac2"}
Sep 10 17:48:19.063620 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.063594    2568 scope.go:117] "RemoveContainer" containerID="8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2"
Sep 10 17:48:19.063910 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:48:19.063891    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2\": container with ID starting with 8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2 not found: ID does not exist" containerID="8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2"
Sep 10 17:48:19.063974 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.063918    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2"} err="failed to get container status \"8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2\": rpc error: code = NotFound desc = could not find container \"8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2\": container with ID starting with 8b7a642dfbce0fd2f44b4adaae6685ce4a47dcb7983c505fefcfa138d6ef00e2 not found: ID does not exist"
Sep 10 17:48:19.063974 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.063935    2568 scope.go:117] "RemoveContainer" containerID="563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027"
Sep 10 17:48:19.071878 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.071858    2568 scope.go:117] "RemoveContainer" containerID="563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027"
Sep 10 17:48:19.072150 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:48:19.072133    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027\": container with ID starting with 563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027 not found: ID does not exist" containerID="563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027"
Sep 10 17:48:19.072203 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.072156    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027"} err="failed to get container status \"563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027\": rpc error: code = NotFound desc = could not find container \"563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027\": container with ID starting with 563339c5f058380114ef5205f04a3c050138b088d4fb3f84d8b190090c523027 not found: ID does not exist"
Sep 10 17:48:19.084348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.084324    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"]
Sep 10 17:48:19.089791 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.089771    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-912593-f5c748bc8-997fg"]
Sep 10 17:48:19.101303 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.101281    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"]
Sep 10 17:48:19.108348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.108325    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-912593-546857d58b-tzfkg"]
Sep 10 17:48:19.764596 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.764557    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f2de142-f1e4-40fc-b076-d6a2f948c82c" path="/var/lib/kubelet/pods/1f2de142-f1e4-40fc-b076-d6a2f948c82c/volumes"
Sep 10 17:48:19.765088 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:19.765073    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4faa207-7ab6-4f7a-a328-ef64d75b8eb1" path="/var/lib/kubelet/pods/a4faa207-7ab6-4f7a-a328-ef64d75b8eb1/volumes"
Sep 10 17:48:34.468839 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.468798    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"]
Sep 10 17:48:34.469512 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469486    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4faa207-7ab6-4f7a-a328-ef64d75b8eb1" containerName="payload-pre-processing"
Sep 10 17:48:34.469690 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469670    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4faa207-7ab6-4f7a-a328-ef64d75b8eb1" containerName="payload-pre-processing"
Sep 10 17:48:34.469785 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469718    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f2de142-f1e4-40fc-b076-d6a2f948c82c" containerName="maas-api"
Sep 10 17:48:34.469785 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469729    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f2de142-f1e4-40fc-b076-d6a2f948c82c" containerName="maas-api"
Sep 10 17:48:34.469785 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469747    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2241f006-2868-4f0a-ae24-7c8490b90ec4" containerName="maas-api"
Sep 10 17:48:34.469785 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469755    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2241f006-2868-4f0a-ae24-7c8490b90ec4" containerName="maas-api"
Sep 10 17:48:34.469917 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469884    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2241f006-2868-4f0a-ae24-7c8490b90ec4" containerName="maas-api"
Sep 10 17:48:34.469917 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469898    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4faa207-7ab6-4f7a-a328-ef64d75b8eb1" containerName="payload-pre-processing"
Sep 10 17:48:34.469917 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.469909    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f2de142-f1e4-40fc-b076-d6a2f948c82c" containerName="maas-api"
Sep 10 17:48:34.473521 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.473501    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.476445 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.476425    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-dc47cd-serving-cert\""
Sep 10 17:48:34.485367 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.485343    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"]
Sep 10 17:48:34.513229 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.513199    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4h7\" (UniqueName: \"kubernetes.io/projected/38c54c62-e5f5-4668-a43b-63d3760ef81b-kube-api-access-mv4h7\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.513438 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.513329    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.513438 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.513365    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-metrics-certs\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.513438 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.513417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c54c62-e5f5-4668-a43b-63d3760ef81b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.614135 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.614097    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.614135 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.614135    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-metrics-certs\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.614430 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.614179    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c54c62-e5f5-4668-a43b-63d3760ef81b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.614430 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.614231    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4h7\" (UniqueName: \"kubernetes.io/projected/38c54c62-e5f5-4668-a43b-63d3760ef81b-kube-api-access-mv4h7\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.614430 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:48:34.614291    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-serving-cert: secret "maas-api-e2e-ipp-a-dc47cd-serving-cert" not found
Sep 10 17:48:34.614430 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:48:34.614389    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls podName:38c54c62-e5f5-4668-a43b-63d3760ef81b nodeName:}" failed. No retries permitted until 2026-09-10 17:48:35.114369869 +0000 UTC m=+1483.823964123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls") pod "maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7" (UID: "38c54c62-e5f5-4668-a43b-63d3760ef81b") : secret "maas-api-e2e-ipp-a-dc47cd-serving-cert" not found
Sep 10 17:48:34.614944 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.614924    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c54c62-e5f5-4668-a43b-63d3760ef81b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.616600 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.616576    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-metrics-certs\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:34.624563 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:34.624542    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4h7\" (UniqueName: \"kubernetes.io/projected/38c54c62-e5f5-4668-a43b-63d3760ef81b-kube-api-access-mv4h7\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:35.118576 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:35.118532    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:35.121081 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:35.121050    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:35.386855 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:35.386763    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:35.659338 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:35.659309    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"]
Sep 10 17:48:35.661388 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:48:35.661362    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c54c62_e5f5_4668_a43b_63d3760ef81b.slice/crio-3f166caab3eab928b28c01a221e75f81a0941974523fd0125d7fac131fa76e92 WatchSource:0}: Error finding container 3f166caab3eab928b28c01a221e75f81a0941974523fd0125d7fac131fa76e92: Status 404 returned error can't find the container with id 3f166caab3eab928b28c01a221e75f81a0941974523fd0125d7fac131fa76e92
Sep 10 17:48:35.979066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:35.978985    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"]
Sep 10 17:48:35.979256 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:35.979212    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk" podUID="c0121041-22a8-4c21-8ad7-da8f9edb32b1" containerName="maas-api" containerID="cri-o://6a035321514a0763cdab6be9c629fe01bf81a3e735ecbcc8d7101e80be683047" gracePeriod=30
Sep 10 17:48:36.035305 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.035275    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"]
Sep 10 17:48:36.035583 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.035494    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw" podUID="35db41c7-3708-4b9b-b78d-1a24ae1a0d1d" containerName="payload-processing" containerID="cri-o://a708568c2565851abf909f9634d81cfe161b300a65434943be432b56831a6b9d" gracePeriod=30
Sep 10 17:48:36.047368 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.046747    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"]
Sep 10 17:48:36.047368 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.046973    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl" podUID="1183b65f-498a-462d-9206-22acb94131fa" containerName="payload-pre-processing" containerID="cri-o://4b886b9b3b9e11968f5008d3de86766e917070c094b6fc1c5aaf8c7c366aa405" gracePeriod=30
Sep 10 17:48:36.169187 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.169152    2568 generic.go:356] "Generic (PLEG): container finished" podID="1183b65f-498a-462d-9206-22acb94131fa" containerID="4b886b9b3b9e11968f5008d3de86766e917070c094b6fc1c5aaf8c7c366aa405" exitCode=0
Sep 10 17:48:36.169352 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.169271    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl" event={"ID":"1183b65f-498a-462d-9206-22acb94131fa","Type":"ContainerDied","Data":"4b886b9b3b9e11968f5008d3de86766e917070c094b6fc1c5aaf8c7c366aa405"}
Sep 10 17:48:36.171689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.171651    2568 generic.go:356] "Generic (PLEG): container finished" podID="35db41c7-3708-4b9b-b78d-1a24ae1a0d1d" containerID="a708568c2565851abf909f9634d81cfe161b300a65434943be432b56831a6b9d" exitCode=0
Sep 10 17:48:36.171820 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.171739    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw" event={"ID":"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d","Type":"ContainerDied","Data":"a708568c2565851abf909f9634d81cfe161b300a65434943be432b56831a6b9d"}
Sep 10 17:48:36.175030 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.174777    2568 generic.go:356] "Generic (PLEG): container finished" podID="c0121041-22a8-4c21-8ad7-da8f9edb32b1" containerID="6a035321514a0763cdab6be9c629fe01bf81a3e735ecbcc8d7101e80be683047" exitCode=0
Sep 10 17:48:36.175030 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.174957    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk" event={"ID":"c0121041-22a8-4c21-8ad7-da8f9edb32b1","Type":"ContainerDied","Data":"6a035321514a0763cdab6be9c629fe01bf81a3e735ecbcc8d7101e80be683047"}
Sep 10 17:48:36.177217 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.177187    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7" event={"ID":"38c54c62-e5f5-4668-a43b-63d3760ef81b","Type":"ContainerStarted","Data":"3f166caab3eab928b28c01a221e75f81a0941974523fd0125d7fac131fa76e92"}
Sep 10 17:48:36.324323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.324296    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:48:36.365552 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.365526    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:48:36.373135 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.373112    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:48:36.432338 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432308    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-plugins-config-volume\") pod \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\" (UID: \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\") "
Sep 10 17:48:36.432532 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432377    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1183b65f-498a-462d-9206-22acb94131fa-plugins-config-volume\") pod \"1183b65f-498a-462d-9206-22acb94131fa\" (UID: \"1183b65f-498a-462d-9206-22acb94131fa\") "
Sep 10 17:48:36.432532 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432507    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-metrics-certs\") pod \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") "
Sep 10 17:48:36.432665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432535    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls\") pod \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") "
Sep 10 17:48:36.432665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432559    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0121041-22a8-4c21-8ad7-da8f9edb32b1-openshift-service-ca\") pod \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") "
Sep 10 17:48:36.432665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432589    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8d8m\" (UniqueName: \"kubernetes.io/projected/c0121041-22a8-4c21-8ad7-da8f9edb32b1-kube-api-access-s8d8m\") pod \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\" (UID: \"c0121041-22a8-4c21-8ad7-da8f9edb32b1\") "
Sep 10 17:48:36.432665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432614    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v2sk\" (UniqueName: \"kubernetes.io/projected/1183b65f-498a-462d-9206-22acb94131fa-kube-api-access-7v2sk\") pod \"1183b65f-498a-462d-9206-22acb94131fa\" (UID: \"1183b65f-498a-462d-9206-22acb94131fa\") "
Sep 10 17:48:36.432862 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432691    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f6g2\" (UniqueName: \"kubernetes.io/projected/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-kube-api-access-2f6g2\") pod \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\" (UID: \"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d\") "
Sep 10 17:48:36.432862 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432734    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35db41c7-3708-4b9b-b78d-1a24ae1a0d1d" (UID: "35db41c7-3708-4b9b-b78d-1a24ae1a0d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:36.432862 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432750    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1183b65f-498a-462d-9206-22acb94131fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1183b65f-498a-462d-9206-22acb94131fa" (UID: "1183b65f-498a-462d-9206-22acb94131fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:36.433023 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.432961    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0121041-22a8-4c21-8ad7-da8f9edb32b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0121041-22a8-4c21-8ad7-da8f9edb32b1" (UID: "c0121041-22a8-4c21-8ad7-da8f9edb32b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:36.433083 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.433021    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:36.433083 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.433045    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1183b65f-498a-462d-9206-22acb94131fa-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:36.434847 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.434816    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0121041-22a8-4c21-8ad7-da8f9edb32b1-kube-api-access-s8d8m" (OuterVolumeSpecName: "kube-api-access-s8d8m") pod "c0121041-22a8-4c21-8ad7-da8f9edb32b1" (UID: "c0121041-22a8-4c21-8ad7-da8f9edb32b1"). InnerVolumeSpecName "kube-api-access-s8d8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:36.434847 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.434830    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0121041-22a8-4c21-8ad7-da8f9edb32b1" (UID: "c0121041-22a8-4c21-8ad7-da8f9edb32b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:48:36.434847 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.434829    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-kube-api-access-2f6g2" (OuterVolumeSpecName: "kube-api-access-2f6g2") pod "35db41c7-3708-4b9b-b78d-1a24ae1a0d1d" (UID: "35db41c7-3708-4b9b-b78d-1a24ae1a0d1d"). InnerVolumeSpecName "kube-api-access-2f6g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:36.435073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.435056    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c0121041-22a8-4c21-8ad7-da8f9edb32b1" (UID: "c0121041-22a8-4c21-8ad7-da8f9edb32b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:48:36.435073 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.435062    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1183b65f-498a-462d-9206-22acb94131fa-kube-api-access-7v2sk" (OuterVolumeSpecName: "kube-api-access-7v2sk") pod "1183b65f-498a-462d-9206-22acb94131fa" (UID: "1183b65f-498a-462d-9206-22acb94131fa"). InnerVolumeSpecName "kube-api-access-7v2sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:36.534095 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.534059    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:36.534095 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.534095    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0121041-22a8-4c21-8ad7-da8f9edb32b1-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:36.534311 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.534108    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0121041-22a8-4c21-8ad7-da8f9edb32b1-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:36.534311 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.534121    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8d8m\" (UniqueName: \"kubernetes.io/projected/c0121041-22a8-4c21-8ad7-da8f9edb32b1-kube-api-access-s8d8m\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:36.534311 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.534134    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v2sk\" (UniqueName: \"kubernetes.io/projected/1183b65f-498a-462d-9206-22acb94131fa-kube-api-access-7v2sk\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:36.534311 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:36.534147    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f6g2\" (UniqueName: \"kubernetes.io/projected/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d-kube-api-access-2f6g2\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:37.185069 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.185033    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7" event={"ID":"38c54c62-e5f5-4668-a43b-63d3760ef81b","Type":"ContainerStarted","Data":"6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f"}
Sep 10 17:48:37.185537 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.185146    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:37.186529 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.186505    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"
Sep 10 17:48:37.186687 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.186511    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl" event={"ID":"1183b65f-498a-462d-9206-22acb94131fa","Type":"ContainerDied","Data":"0db5764773c6894cc86e69628e1c46fddcd68b6eb9f2c552206d2c46100c874f"}
Sep 10 17:48:37.186687 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.186633    2568 scope.go:117] "RemoveContainer" containerID="4b886b9b3b9e11968f5008d3de86766e917070c094b6fc1c5aaf8c7c366aa405"
Sep 10 17:48:37.188055 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.187865    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw" event={"ID":"35db41c7-3708-4b9b-b78d-1a24ae1a0d1d","Type":"ContainerDied","Data":"7f56784a42619887b8b68ddf2f43b01e5a58dcab48c2100208789187a3d5774d"}
Sep 10 17:48:37.188055 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.187889    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"
Sep 10 17:48:37.189383 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.189361    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk" event={"ID":"c0121041-22a8-4c21-8ad7-da8f9edb32b1","Type":"ContainerDied","Data":"4d809f7d5da99dc1cfbe355c92280e8fbc5effb4ab821608a1d32fb7ad22fb98"}
Sep 10 17:48:37.189481 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.189368    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"
Sep 10 17:48:37.197698 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.197675    2568 scope.go:117] "RemoveContainer" containerID="a708568c2565851abf909f9634d81cfe161b300a65434943be432b56831a6b9d"
Sep 10 17:48:37.204416 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.204376    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7" podStartSLOduration=2.733547123 podStartE2EDuration="3.204362785s" totalImagesPullingTime="470.815662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:37.202805817 +0000 UTC m=+1485.912400083" watchObservedRunningTime="2026-09-10 17:48:37.204362785 +0000 UTC m=+1485.913957051"
Sep 10 17:48:37.207479 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.207462    2568 scope.go:117] "RemoveContainer" containerID="6a035321514a0763cdab6be9c629fe01bf81a3e735ecbcc8d7101e80be683047"
Sep 10 17:48:37.220765 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.220742    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"]
Sep 10 17:48:37.224525 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.224505    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c990a5-658fd9c97d-xsmvl"]
Sep 10 17:48:37.235528 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.235495    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"]
Sep 10 17:48:37.240062 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.240036    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c990a5-6b6958b96c-t2dtw"]
Sep 10 17:48:37.250596 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.250573    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"]
Sep 10 17:48:37.254445 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.254418    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c990a5-5799c85f4b-xzkdk"]
Sep 10 17:48:37.764585 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.764554    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1183b65f-498a-462d-9206-22acb94131fa" path="/var/lib/kubelet/pods/1183b65f-498a-462d-9206-22acb94131fa/volumes"
Sep 10 17:48:37.764911 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.764899    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35db41c7-3708-4b9b-b78d-1a24ae1a0d1d" path="/var/lib/kubelet/pods/35db41c7-3708-4b9b-b78d-1a24ae1a0d1d/volumes"
Sep 10 17:48:37.765268 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:37.765224    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0121041-22a8-4c21-8ad7-da8f9edb32b1" path="/var/lib/kubelet/pods/c0121041-22a8-4c21-8ad7-da8f9edb32b1/volumes"
Sep 10 17:48:43.202887 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:43.202854    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:48:51.815689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:51.815658    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:48:51.819023 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:51.819002    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:48:53.781187 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:53.781154    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"]
Sep 10 17:48:53.781604 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:53.781428    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25" podUID="7e17b7ce-6430-4afc-9968-06c1f09f55d1" containerName="maas-api" containerID="cri-o://4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec" gracePeriod=30
Sep 10 17:48:53.855283 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:53.855249    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"]
Sep 10 17:48:54.039527 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.039473    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:48:54.199876 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.199843    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjkkn\" (UniqueName: \"kubernetes.io/projected/7e17b7ce-6430-4afc-9968-06c1f09f55d1-kube-api-access-fjkkn\") pod \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") "
Sep 10 17:48:54.200066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.199885    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls\") pod \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") "
Sep 10 17:48:54.200066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.199923    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e17b7ce-6430-4afc-9968-06c1f09f55d1-openshift-service-ca\") pod \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") "
Sep 10 17:48:54.200066 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.199967    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-metrics-certs\") pod \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\" (UID: \"7e17b7ce-6430-4afc-9968-06c1f09f55d1\") "
Sep 10 17:48:54.200295 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.200276    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e17b7ce-6430-4afc-9968-06c1f09f55d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e17b7ce-6430-4afc-9968-06c1f09f55d1" (UID: "7e17b7ce-6430-4afc-9968-06c1f09f55d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:54.202021 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.201998    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e17b7ce-6430-4afc-9968-06c1f09f55d1" (UID: "7e17b7ce-6430-4afc-9968-06c1f09f55d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:48:54.202128 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.202043    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e17b7ce-6430-4afc-9968-06c1f09f55d1-kube-api-access-fjkkn" (OuterVolumeSpecName: "kube-api-access-fjkkn") pod "7e17b7ce-6430-4afc-9968-06c1f09f55d1" (UID: "7e17b7ce-6430-4afc-9968-06c1f09f55d1"). InnerVolumeSpecName "kube-api-access-fjkkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:54.202128 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.202102    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7e17b7ce-6430-4afc-9968-06c1f09f55d1" (UID: "7e17b7ce-6430-4afc-9968-06c1f09f55d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:48:54.283373 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.283331    2568 generic.go:356] "Generic (PLEG): container finished" podID="7e17b7ce-6430-4afc-9968-06c1f09f55d1" containerID="4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec" exitCode=0
Sep 10 17:48:54.283587 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.283386    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"
Sep 10 17:48:54.283587 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.283426    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25" event={"ID":"7e17b7ce-6430-4afc-9968-06c1f09f55d1","Type":"ContainerDied","Data":"4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec"}
Sep 10 17:48:54.283587 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.283473    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25" event={"ID":"7e17b7ce-6430-4afc-9968-06c1f09f55d1","Type":"ContainerDied","Data":"d830cff70fac3b56062ef2bfc04040972594463f76bf61111da195f4f9711e4e"}
Sep 10 17:48:54.283587 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.283494    2568 scope.go:117] "RemoveContainer" containerID="4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec"
Sep 10 17:48:54.283811 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.283668    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7" podUID="ef3f1703-d25d-4cec-a6a4-824da7a9e221" containerName="payload-processing" containerID="cri-o://c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688" gracePeriod=30
Sep 10 17:48:54.293165 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.293142    2568 scope.go:117] "RemoveContainer" containerID="4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec"
Sep 10 17:48:54.293521 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:48:54.293497    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec\": container with ID starting with 4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec not found: ID does not exist" containerID="4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec"
Sep 10 17:48:54.293621 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.293528    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec"} err="failed to get container status \"4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec\": rpc error: code = NotFound desc = could not find container \"4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec\": container with ID starting with 4b71deea741df3a026dac583e810c4fe6d52d59c2685c5f50e64c5dd8263f2ec not found: ID does not exist"
Sep 10 17:48:54.300963 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.300938    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e17b7ce-6430-4afc-9968-06c1f09f55d1-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:54.300963 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.300963    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:54.301100 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.300974    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkkn\" (UniqueName: \"kubernetes.io/projected/7e17b7ce-6430-4afc-9968-06c1f09f55d1-kube-api-access-fjkkn\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:54.301100 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.300983    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e17b7ce-6430-4afc-9968-06c1f09f55d1-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:54.318124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.318097    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"]
Sep 10 17:48:54.321959 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.321935    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-762738-6649ddcdbc-tkf25"]
Sep 10 17:48:54.518258 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.518218    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:48:54.603124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.603039    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3f1703-d25d-4cec-a6a4-824da7a9e221-plugins-config-volume\") pod \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\" (UID: \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\") "
Sep 10 17:48:54.603124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.603101    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/ef3f1703-d25d-4cec-a6a4-824da7a9e221-kube-api-access-w2rf6\") pod \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\" (UID: \"ef3f1703-d25d-4cec-a6a4-824da7a9e221\") "
Sep 10 17:48:54.603473 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.603449    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3f1703-d25d-4cec-a6a4-824da7a9e221-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef3f1703-d25d-4cec-a6a4-824da7a9e221" (UID: "ef3f1703-d25d-4cec-a6a4-824da7a9e221"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:48:54.605126 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.605107    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3f1703-d25d-4cec-a6a4-824da7a9e221-kube-api-access-w2rf6" (OuterVolumeSpecName: "kube-api-access-w2rf6") pod "ef3f1703-d25d-4cec-a6a4-824da7a9e221" (UID: "ef3f1703-d25d-4cec-a6a4-824da7a9e221"). InnerVolumeSpecName "kube-api-access-w2rf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:48:54.703978 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.703946    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef3f1703-d25d-4cec-a6a4-824da7a9e221-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:54.703978 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:54.703974    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/ef3f1703-d25d-4cec-a6a4-824da7a9e221-kube-api-access-w2rf6\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:48:55.290091 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.290055    2568 generic.go:356] "Generic (PLEG): container finished" podID="ef3f1703-d25d-4cec-a6a4-824da7a9e221" containerID="c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688" exitCode=0
Sep 10 17:48:55.290578 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.290125    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"
Sep 10 17:48:55.290578 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.290144    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7" event={"ID":"ef3f1703-d25d-4cec-a6a4-824da7a9e221","Type":"ContainerDied","Data":"c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688"}
Sep 10 17:48:55.290578 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.290202    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7" event={"ID":"ef3f1703-d25d-4cec-a6a4-824da7a9e221","Type":"ContainerDied","Data":"f2dd74e9c0ce550cbc57a9d6039551d718eead46caada28313bacfabba1d3984"}
Sep 10 17:48:55.290578 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.290247    2568 scope.go:117] "RemoveContainer" containerID="c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688"
Sep 10 17:48:55.301019 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.300983    2568 scope.go:117] "RemoveContainer" containerID="c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688"
Sep 10 17:48:55.301433 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:48:55.301411    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688\": container with ID starting with c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688 not found: ID does not exist" containerID="c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688"
Sep 10 17:48:55.301525 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.301443    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688"} err="failed to get container status \"c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688\": rpc error: code = NotFound desc = could not find container \"c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688\": container with ID starting with c3bfbdbd2cdfffc96091b716048fe78a16d767ced57dd9c65efd34a00ea7f688 not found: ID does not exist"
Sep 10 17:48:55.312725 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.312698    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"]
Sep 10 17:48:55.315471 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.315450    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-762738-5f6c6c4f77-s85h7"]
Sep 10 17:48:55.765188 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.765092    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e17b7ce-6430-4afc-9968-06c1f09f55d1" path="/var/lib/kubelet/pods/7e17b7ce-6430-4afc-9968-06c1f09f55d1/volumes"
Sep 10 17:48:55.765571 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:48:55.765551    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef3f1703-d25d-4cec-a6a4-824da7a9e221" path="/var/lib/kubelet/pods/ef3f1703-d25d-4cec-a6a4-824da7a9e221/volumes"
Sep 10 17:49:03.904573 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.904537    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"]
Sep 10 17:49:03.905208 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905187    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e17b7ce-6430-4afc-9968-06c1f09f55d1" containerName="maas-api"
Sep 10 17:49:03.905310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905211    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e17b7ce-6430-4afc-9968-06c1f09f55d1" containerName="maas-api"
Sep 10 17:49:03.905310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905224    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35db41c7-3708-4b9b-b78d-1a24ae1a0d1d" containerName="payload-processing"
Sep 10 17:49:03.905310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905248    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="35db41c7-3708-4b9b-b78d-1a24ae1a0d1d" containerName="payload-processing"
Sep 10 17:49:03.905310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905270    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1183b65f-498a-462d-9206-22acb94131fa" containerName="payload-pre-processing"
Sep 10 17:49:03.905310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905279    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1183b65f-498a-462d-9206-22acb94131fa" containerName="payload-pre-processing"
Sep 10 17:49:03.905310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905295    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0121041-22a8-4c21-8ad7-da8f9edb32b1" containerName="maas-api"
Sep 10 17:49:03.905310 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905305    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0121041-22a8-4c21-8ad7-da8f9edb32b1" containerName="maas-api"
Sep 10 17:49:03.905560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905315    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3f1703-d25d-4cec-a6a4-824da7a9e221" containerName="payload-processing"
Sep 10 17:49:03.905560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905324    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3f1703-d25d-4cec-a6a4-824da7a9e221" containerName="payload-processing"
Sep 10 17:49:03.905560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905418    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="35db41c7-3708-4b9b-b78d-1a24ae1a0d1d" containerName="payload-processing"
Sep 10 17:49:03.905560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905431    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0121041-22a8-4c21-8ad7-da8f9edb32b1" containerName="maas-api"
Sep 10 17:49:03.905560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905443    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3f1703-d25d-4cec-a6a4-824da7a9e221" containerName="payload-processing"
Sep 10 17:49:03.905560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905453    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1183b65f-498a-462d-9206-22acb94131fa" containerName="payload-pre-processing"
Sep 10 17:49:03.905560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.905462    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e17b7ce-6430-4afc-9968-06c1f09f55d1" containerName="maas-api"
Sep 10 17:49:03.910303 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.910278    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:03.913520 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.913154    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7621a1-serving-cert\""
Sep 10 17:49:03.919915 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:03.919668    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"]
Sep 10 17:49:04.089247 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.089205    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0d88a-fd3a-4c65-9385-8f12495033b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.089432 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.089331    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-metrics-certs\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.089432 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.089399    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p6gv\" (UniqueName: \"kubernetes.io/projected/2ef0d88a-fd3a-4c65-9385-8f12495033b8-kube-api-access-5p6gv\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.089518 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.089434    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.190126 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.190040    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.190126 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.190103    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0d88a-fd3a-4c65-9385-8f12495033b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.190442 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.190149    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-metrics-certs\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.190442 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:49:04.190177    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-serving-cert: secret "maas-api-e2e-ipp-b-7621a1-serving-cert" not found
Sep 10 17:49:04.190442 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.190196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p6gv\" (UniqueName: \"kubernetes.io/projected/2ef0d88a-fd3a-4c65-9385-8f12495033b8-kube-api-access-5p6gv\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.190442 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:49:04.190277    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls podName:2ef0d88a-fd3a-4c65-9385-8f12495033b8 nodeName:}" failed. No retries permitted until 2026-09-10 17:49:04.690228226 +0000 UTC m=+1513.399822470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls") pod "maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6" (UID: "2ef0d88a-fd3a-4c65-9385-8f12495033b8") : secret "maas-api-e2e-ipp-b-7621a1-serving-cert" not found
Sep 10 17:49:04.190870 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.190844    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0d88a-fd3a-4c65-9385-8f12495033b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.192721 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.192697    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-metrics-certs\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.198820 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.198797    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p6gv\" (UniqueName: \"kubernetes.io/projected/2ef0d88a-fd3a-4c65-9385-8f12495033b8-kube-api-access-5p6gv\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.539712 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.539672    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"]
Sep 10 17:49:04.543261 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.543214    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:04.546025 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.545995    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7621a1\""
Sep 10 17:49:04.546126 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.546060    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7621a1-dockercfg-58hf9\""
Sep 10 17:49:04.553951 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.553921    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"]
Sep 10 17:49:04.565365 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.565334    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"]
Sep 10 17:49:04.569072 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.569033    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:04.593493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.593462    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq\" (UID: \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:04.593656 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.593554    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d04d00-794f-47c6-9e0f-ef4a171d153a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg\" (UID: \"67d04d00-794f-47c6-9e0f-ef4a171d153a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:04.593656 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.593585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzlx\" (UniqueName: \"kubernetes.io/projected/67d04d00-794f-47c6-9e0f-ef4a171d153a-kube-api-access-4jzlx\") pod \"payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg\" (UID: \"67d04d00-794f-47c6-9e0f-ef4a171d153a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:04.593656 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.593637    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9c2\" (UniqueName: \"kubernetes.io/projected/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-kube-api-access-zq9c2\") pod \"payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq\" (UID: \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:04.596000 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.595976    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"]
Sep 10 17:49:04.694615 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.694572    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzlx\" (UniqueName: \"kubernetes.io/projected/67d04d00-794f-47c6-9e0f-ef4a171d153a-kube-api-access-4jzlx\") pod \"payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg\" (UID: \"67d04d00-794f-47c6-9e0f-ef4a171d153a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:04.694814 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.694630    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9c2\" (UniqueName: \"kubernetes.io/projected/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-kube-api-access-zq9c2\") pod \"payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq\" (UID: \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:04.694814 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.694667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.694814 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.694696    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq\" (UID: \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:04.694814 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.694748    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d04d00-794f-47c6-9e0f-ef4a171d153a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg\" (UID: \"67d04d00-794f-47c6-9e0f-ef4a171d153a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:04.695493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.695470    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d04d00-794f-47c6-9e0f-ef4a171d153a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg\" (UID: \"67d04d00-794f-47c6-9e0f-ef4a171d153a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:04.695493 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.695480    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq\" (UID: \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:04.697053 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.697034    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.702990 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.702964    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzlx\" (UniqueName: \"kubernetes.io/projected/67d04d00-794f-47c6-9e0f-ef4a171d153a-kube-api-access-4jzlx\") pod \"payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg\" (UID: \"67d04d00-794f-47c6-9e0f-ef4a171d153a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:04.703332 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.703312    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9c2\" (UniqueName: \"kubernetes.io/projected/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-kube-api-access-zq9c2\") pod \"payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq\" (UID: \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:04.823932 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.823845    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:04.856889 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.856857    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:04.881658 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:04.880973    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:05.106166 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.106140    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"]
Sep 10 17:49:05.108049 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:49:05.108023    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef0d88a_fd3a_4c65_9385_8f12495033b8.slice/crio-62feee123cdc4c80dedfa6af68929765df4a2b5d18762af50a18a52dbb3ca157 WatchSource:0}: Error finding container 62feee123cdc4c80dedfa6af68929765df4a2b5d18762af50a18a52dbb3ca157: Status 404 returned error can't find the container with id 62feee123cdc4c80dedfa6af68929765df4a2b5d18762af50a18a52dbb3ca157
Sep 10 17:49:05.144845 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.144812    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"]
Sep 10 17:49:05.146428 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:49:05.146395    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d04d00_794f_47c6_9e0f_ef4a171d153a.slice/crio-151a8391c09edcc0af352291d7ce1d795a30000ecb77087179c16ddb48c360dc WatchSource:0}: Error finding container 151a8391c09edcc0af352291d7ce1d795a30000ecb77087179c16ddb48c360dc: Status 404 returned error can't find the container with id 151a8391c09edcc0af352291d7ce1d795a30000ecb77087179c16ddb48c360dc
Sep 10 17:49:05.177816 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.177772    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"]
Sep 10 17:49:05.347139 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.347097    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq" event={"ID":"5ecf1b1b-854e-47d6-a199-5c4a516beb8c","Type":"ContainerStarted","Data":"dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3"}
Sep 10 17:49:05.347368 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.347178    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq" event={"ID":"5ecf1b1b-854e-47d6-a199-5c4a516beb8c","Type":"ContainerStarted","Data":"72bab4f5e4360befa64db58a526c1ce2ef3d8671def50e10e2688ed705812e49"}
Sep 10 17:49:05.347368 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.347332    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:05.348961 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.348933    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6" event={"ID":"2ef0d88a-fd3a-4c65-9385-8f12495033b8","Type":"ContainerStarted","Data":"62feee123cdc4c80dedfa6af68929765df4a2b5d18762af50a18a52dbb3ca157"}
Sep 10 17:49:05.350990 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.350955    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg" event={"ID":"67d04d00-794f-47c6-9e0f-ef4a171d153a","Type":"ContainerStarted","Data":"889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f"}
Sep 10 17:49:05.351107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.351004    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg" event={"ID":"67d04d00-794f-47c6-9e0f-ef4a171d153a","Type":"ContainerStarted","Data":"151a8391c09edcc0af352291d7ce1d795a30000ecb77087179c16ddb48c360dc"}
Sep 10 17:49:05.351107 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.351032    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:05.364676 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.364594    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq" podStartSLOduration=1.364580068 podStartE2EDuration="1.364580068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:05.36374292 +0000 UTC m=+1514.073337189" watchObservedRunningTime="2026-09-10 17:49:05.364580068 +0000 UTC m=+1514.074174334"
Sep 10 17:49:05.382408 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:05.382346    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg" podStartSLOduration=1.382330932 podStartE2EDuration="1.382330932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:05.380705344 +0000 UTC m=+1514.090299613" watchObservedRunningTime="2026-09-10 17:49:05.382330932 +0000 UTC m=+1514.091925198"
Sep 10 17:49:06.358694 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:06.358653    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6" event={"ID":"2ef0d88a-fd3a-4c65-9385-8f12495033b8","Type":"ContainerStarted","Data":"cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16"}
Sep 10 17:49:06.359170 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:06.358832    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:06.378151 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:06.378103    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6" podStartSLOduration=3.017462643 podStartE2EDuration="3.378088726s" totalImagesPullingTime="360.626083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:49:06.375905828 +0000 UTC m=+1515.085500095" watchObservedRunningTime="2026-09-10 17:49:06.378088726 +0000 UTC m=+1515.087682989"
Sep 10 17:49:12.369347 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:12.369315    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:49:16.361304 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:16.361203    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:49:16.361662 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:16.361637    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:49:32.863925 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:49:32.863891    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq_5ecf1b1b-854e-47d6-a199-5c4a516beb8c/payload-processing/0.log"
Sep 10 17:50:05.920470 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:05.920436    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq_5ecf1b1b-854e-47d6-a199-5c4a516beb8c/payload-processing/0.log"
Sep 10 17:50:11.186118 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.186081    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"]
Sep 10 17:50:11.190223 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.190206    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.193064 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.193044    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8e4477-serving-cert\""
Sep 10 17:50:11.205127 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.205100    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"]
Sep 10 17:50:11.275545 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.275510    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb275291-d739-4585-a85f-c28436c69cf3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.275740 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.275557    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w79v7\" (UniqueName: \"kubernetes.io/projected/eb275291-d739-4585-a85f-c28436c69cf3-kube-api-access-w79v7\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.275740 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.275592    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.275740 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.275717    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.376347 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.376312    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb275291-d739-4585-a85f-c28436c69cf3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.376520 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.376359    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w79v7\" (UniqueName: \"kubernetes.io/projected/eb275291-d739-4585-a85f-c28436c69cf3-kube-api-access-w79v7\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.376520 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.376383    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.376520 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.376430    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.376520 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:50:11.376511    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-serving-cert: secret "maas-api-e2e-ipp-cleanup-8e4477-serving-cert" not found
Sep 10 17:50:11.376663 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:50:11.376570    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls podName:eb275291-d739-4585-a85f-c28436c69cf3 nodeName:}" failed. No retries permitted until 2026-09-10 17:50:11.876555088 +0000 UTC m=+1580.586149332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl" (UID: "eb275291-d739-4585-a85f-c28436c69cf3") : secret "maas-api-e2e-ipp-cleanup-8e4477-serving-cert" not found
Sep 10 17:50:11.376960 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.376936    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb275291-d739-4585-a85f-c28436c69cf3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.378820 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.378797    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.386960 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.386931    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w79v7\" (UniqueName: \"kubernetes.io/projected/eb275291-d739-4585-a85f-c28436c69cf3-kube-api-access-w79v7\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.880592 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.880542    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.882532 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.882498    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"]
Sep 10 17:50:11.883271 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.883231    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:11.891675 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.891646    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:11.896174 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.896151    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8e4477-dockercfg-8nh5g\""
Sep 10 17:50:11.896287 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.896155    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8e4477\""
Sep 10 17:50:11.904749 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.904724    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"]
Sep 10 17:50:11.981551 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.981521    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f24ee3-3cfc-424d-86ff-345627abd310-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg\" (UID: \"48f24ee3-3cfc-424d-86ff-345627abd310\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:11.981755 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:11.981658    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dq8c\" (UniqueName: \"kubernetes.io/projected/48f24ee3-3cfc-424d-86ff-345627abd310-kube-api-access-7dq8c\") pod \"payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg\" (UID: \"48f24ee3-3cfc-424d-86ff-345627abd310\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:12.082773 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.082727    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f24ee3-3cfc-424d-86ff-345627abd310-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg\" (UID: \"48f24ee3-3cfc-424d-86ff-345627abd310\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:12.082999 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.082856    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dq8c\" (UniqueName: \"kubernetes.io/projected/48f24ee3-3cfc-424d-86ff-345627abd310-kube-api-access-7dq8c\") pod \"payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg\" (UID: \"48f24ee3-3cfc-424d-86ff-345627abd310\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:12.083523 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.083500    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f24ee3-3cfc-424d-86ff-345627abd310-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg\" (UID: \"48f24ee3-3cfc-424d-86ff-345627abd310\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:12.092043 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.092008    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dq8c\" (UniqueName: \"kubernetes.io/projected/48f24ee3-3cfc-424d-86ff-345627abd310-kube-api-access-7dq8c\") pod \"payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg\" (UID: \"48f24ee3-3cfc-424d-86ff-345627abd310\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:12.103026 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.103004    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:12.205051 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.205014    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:12.381867 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.381841    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"]
Sep 10 17:50:12.383605 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:50:12.383576    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb275291_d739_4585_a85f_c28436c69cf3.slice/crio-f56f0cc7a4ae6117cab9d8d060f10b1592e4aa45f916fff657a5d4aadf8a0a8c WatchSource:0}: Error finding container f56f0cc7a4ae6117cab9d8d060f10b1592e4aa45f916fff657a5d4aadf8a0a8c: Status 404 returned error can't find the container with id f56f0cc7a4ae6117cab9d8d060f10b1592e4aa45f916fff657a5d4aadf8a0a8c
Sep 10 17:50:12.478291 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.478264    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"]
Sep 10 17:50:12.480276 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:50:12.480217    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f24ee3_3cfc_424d_86ff_345627abd310.slice/crio-438f7f4f2cd8b247e7cf349a53753e2e1b277a8338f447445fcc97037f0698b0 WatchSource:0}: Error finding container 438f7f4f2cd8b247e7cf349a53753e2e1b277a8338f447445fcc97037f0698b0: Status 404 returned error can't find the container with id 438f7f4f2cd8b247e7cf349a53753e2e1b277a8338f447445fcc97037f0698b0
Sep 10 17:50:12.697213 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.697173    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg" event={"ID":"48f24ee3-3cfc-424d-86ff-345627abd310","Type":"ContainerStarted","Data":"7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc"}
Sep 10 17:50:12.697413 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.697250    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg" event={"ID":"48f24ee3-3cfc-424d-86ff-345627abd310","Type":"ContainerStarted","Data":"438f7f4f2cd8b247e7cf349a53753e2e1b277a8338f447445fcc97037f0698b0"}
Sep 10 17:50:12.697413 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.697375    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:12.698480 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.698454    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl" event={"ID":"eb275291-d739-4585-a85f-c28436c69cf3","Type":"ContainerStarted","Data":"f56f0cc7a4ae6117cab9d8d060f10b1592e4aa45f916fff657a5d4aadf8a0a8c"}
Sep 10 17:50:12.716327 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:12.716281    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg" podStartSLOduration=1.716266515 podStartE2EDuration="1.716266515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:12.714603118 +0000 UTC m=+1581.424197385" watchObservedRunningTime="2026-09-10 17:50:12.716266515 +0000 UTC m=+1581.425860780"
Sep 10 17:50:13.704605 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:13.704568    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl" event={"ID":"eb275291-d739-4585-a85f-c28436c69cf3","Type":"ContainerStarted","Data":"bb9a6d574ed31c76bff154fa0a3229a52ff3041ef52b246370a4619c3b4c5802"}
Sep 10 17:50:13.704994 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:13.704804    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:13.722587 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:13.722537    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl" podStartSLOduration=2.376819141 podStartE2EDuration="2.722523195s" totalImagesPullingTime="345.704054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:13.721659028 +0000 UTC m=+1582.431253307" watchObservedRunningTime="2026-09-10 17:50:13.722523195 +0000 UTC m=+1582.432117460"
Sep 10 17:50:19.715619 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:19.715588    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:23.706968 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:23.706935    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:41.714085 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:41.714054    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"]
Sep 10 17:50:41.714587 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:41.714305    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl" podUID="eb275291-d739-4585-a85f-c28436c69cf3" containerName="maas-api" containerID="cri-o://bb9a6d574ed31c76bff154fa0a3229a52ff3041ef52b246370a4619c3b4c5802" gracePeriod=30
Sep 10 17:50:41.774692 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:41.774659    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"]
Sep 10 17:50:41.774910 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:41.774884    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg" podUID="48f24ee3-3cfc-424d-86ff-345627abd310" containerName="payload-processing" containerID="cri-o://7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc" gracePeriod=30
Sep 10 17:50:41.854662 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:41.854625    2568 generic.go:356] "Generic (PLEG): container finished" podID="eb275291-d739-4585-a85f-c28436c69cf3" containerID="bb9a6d574ed31c76bff154fa0a3229a52ff3041ef52b246370a4619c3b4c5802" exitCode=0
Sep 10 17:50:41.854861 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:41.854702    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl" event={"ID":"eb275291-d739-4585-a85f-c28436c69cf3","Type":"ContainerDied","Data":"bb9a6d574ed31c76bff154fa0a3229a52ff3041ef52b246370a4619c3b4c5802"}
Sep 10 17:50:42.022414 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.022389    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:42.053665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.053640    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:42.063134 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.063102    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dq8c\" (UniqueName: \"kubernetes.io/projected/48f24ee3-3cfc-424d-86ff-345627abd310-kube-api-access-7dq8c\") pod \"48f24ee3-3cfc-424d-86ff-345627abd310\" (UID: \"48f24ee3-3cfc-424d-86ff-345627abd310\") "
Sep 10 17:50:42.063279 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.063183    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls\") pod \"eb275291-d739-4585-a85f-c28436c69cf3\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") "
Sep 10 17:50:42.063358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.063297    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb275291-d739-4585-a85f-c28436c69cf3-openshift-service-ca\") pod \"eb275291-d739-4585-a85f-c28436c69cf3\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") "
Sep 10 17:50:42.063358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.063324    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-metrics-certs\") pod \"eb275291-d739-4585-a85f-c28436c69cf3\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") "
Sep 10 17:50:42.063469 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.063387    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w79v7\" (UniqueName: \"kubernetes.io/projected/eb275291-d739-4585-a85f-c28436c69cf3-kube-api-access-w79v7\") pod \"eb275291-d739-4585-a85f-c28436c69cf3\" (UID: \"eb275291-d739-4585-a85f-c28436c69cf3\") "
Sep 10 17:50:42.063469 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.063417    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f24ee3-3cfc-424d-86ff-345627abd310-plugins-config-volume\") pod \"48f24ee3-3cfc-424d-86ff-345627abd310\" (UID: \"48f24ee3-3cfc-424d-86ff-345627abd310\") "
Sep 10 17:50:42.063809 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.063690    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb275291-d739-4585-a85f-c28436c69cf3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb275291-d739-4585-a85f-c28436c69cf3" (UID: "eb275291-d739-4585-a85f-c28436c69cf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:42.063920 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.063892    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f24ee3-3cfc-424d-86ff-345627abd310-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48f24ee3-3cfc-424d-86ff-345627abd310" (UID: "48f24ee3-3cfc-424d-86ff-345627abd310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:50:42.065749 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.065724    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb275291-d739-4585-a85f-c28436c69cf3" (UID: "eb275291-d739-4585-a85f-c28436c69cf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:50:42.065984 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.065956    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f24ee3-3cfc-424d-86ff-345627abd310-kube-api-access-7dq8c" (OuterVolumeSpecName: "kube-api-access-7dq8c") pod "48f24ee3-3cfc-424d-86ff-345627abd310" (UID: "48f24ee3-3cfc-424d-86ff-345627abd310"). InnerVolumeSpecName "kube-api-access-7dq8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:42.066175 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.066150    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eb275291-d739-4585-a85f-c28436c69cf3" (UID: "eb275291-d739-4585-a85f-c28436c69cf3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:50:42.066292 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.066206    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb275291-d739-4585-a85f-c28436c69cf3-kube-api-access-w79v7" (OuterVolumeSpecName: "kube-api-access-w79v7") pod "eb275291-d739-4585-a85f-c28436c69cf3" (UID: "eb275291-d739-4585-a85f-c28436c69cf3"). InnerVolumeSpecName "kube-api-access-w79v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:50:42.165027 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.164993    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w79v7\" (UniqueName: \"kubernetes.io/projected/eb275291-d739-4585-a85f-c28436c69cf3-kube-api-access-w79v7\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:50:42.165027 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.165021    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f24ee3-3cfc-424d-86ff-345627abd310-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:50:42.165027 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.165030    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dq8c\" (UniqueName: \"kubernetes.io/projected/48f24ee3-3cfc-424d-86ff-345627abd310-kube-api-access-7dq8c\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:50:42.165280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.165039    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:50:42.165280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.165049    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb275291-d739-4585-a85f-c28436c69cf3-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:50:42.165280 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.165057    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb275291-d739-4585-a85f-c28436c69cf3-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:50:42.862225 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.862191    2568 generic.go:356] "Generic (PLEG): container finished" podID="48f24ee3-3cfc-424d-86ff-345627abd310" containerID="7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc" exitCode=0
Sep 10 17:50:42.862679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.862252    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"
Sep 10 17:50:42.862679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.862277    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg" event={"ID":"48f24ee3-3cfc-424d-86ff-345627abd310","Type":"ContainerDied","Data":"7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc"}
Sep 10 17:50:42.862679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.862347    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg" event={"ID":"48f24ee3-3cfc-424d-86ff-345627abd310","Type":"ContainerDied","Data":"438f7f4f2cd8b247e7cf349a53753e2e1b277a8338f447445fcc97037f0698b0"}
Sep 10 17:50:42.862679 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.862385    2568 scope.go:117] "RemoveContainer" containerID="7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc"
Sep 10 17:50:42.863903 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.863785    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl" event={"ID":"eb275291-d739-4585-a85f-c28436c69cf3","Type":"ContainerDied","Data":"f56f0cc7a4ae6117cab9d8d060f10b1592e4aa45f916fff657a5d4aadf8a0a8c"}
Sep 10 17:50:42.863903 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.863806    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"
Sep 10 17:50:42.871732 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.871713    2568 scope.go:117] "RemoveContainer" containerID="7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc"
Sep 10 17:50:42.872058 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:50:42.872039    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc\": container with ID starting with 7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc not found: ID does not exist" containerID="7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc"
Sep 10 17:50:42.872156 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.872067    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc"} err="failed to get container status \"7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc\": rpc error: code = NotFound desc = could not find container \"7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc\": container with ID starting with 7108fe2642adce5b0d8823a910bcb719e203162f9d9f23211bc7ce0760cff3fc not found: ID does not exist"
Sep 10 17:50:42.872156 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.872090    2568 scope.go:117] "RemoveContainer" containerID="bb9a6d574ed31c76bff154fa0a3229a52ff3041ef52b246370a4619c3b4c5802"
Sep 10 17:50:42.891439 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.891417    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"]
Sep 10 17:50:42.899165 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.899143    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e4477-75f5688f65-2qvqg"]
Sep 10 17:50:42.911629 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.911608    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"]
Sep 10 17:50:42.915447 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:42.915423    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8e4477-6448f5455c-8ccpl"]
Sep 10 17:50:43.765638 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:43.765608    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f24ee3-3cfc-424d-86ff-345627abd310" path="/var/lib/kubelet/pods/48f24ee3-3cfc-424d-86ff-345627abd310/volumes"
Sep 10 17:50:43.766024 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:50:43.766000    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb275291-d739-4585-a85f-c28436c69cf3" path="/var/lib/kubelet/pods/eb275291-d739-4585-a85f-c28436c69cf3/volumes"
Sep 10 17:51:07.883726 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:07.883694    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"]
Sep 10 17:51:07.884275 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:07.883898    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7" podUID="38c54c62-e5f5-4668-a43b-63d3760ef81b" containerName="maas-api" containerID="cri-o://6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f" gracePeriod=30
Sep 10 17:51:08.134910 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.134851    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:51:08.307867 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.307835    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv4h7\" (UniqueName: \"kubernetes.io/projected/38c54c62-e5f5-4668-a43b-63d3760ef81b-kube-api-access-mv4h7\") pod \"38c54c62-e5f5-4668-a43b-63d3760ef81b\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") "
Sep 10 17:51:08.307867 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.307870    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls\") pod \"38c54c62-e5f5-4668-a43b-63d3760ef81b\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") "
Sep 10 17:51:08.308124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.307893    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c54c62-e5f5-4668-a43b-63d3760ef81b-openshift-service-ca\") pod \"38c54c62-e5f5-4668-a43b-63d3760ef81b\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") "
Sep 10 17:51:08.308124 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.307993    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-metrics-certs\") pod \"38c54c62-e5f5-4668-a43b-63d3760ef81b\" (UID: \"38c54c62-e5f5-4668-a43b-63d3760ef81b\") "
Sep 10 17:51:08.308343 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.308311    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c54c62-e5f5-4668-a43b-63d3760ef81b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38c54c62-e5f5-4668-a43b-63d3760ef81b" (UID: "38c54c62-e5f5-4668-a43b-63d3760ef81b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:51:08.310077 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.310053    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "38c54c62-e5f5-4668-a43b-63d3760ef81b" (UID: "38c54c62-e5f5-4668-a43b-63d3760ef81b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:51:08.310162 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.310095    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c54c62-e5f5-4668-a43b-63d3760ef81b-kube-api-access-mv4h7" (OuterVolumeSpecName: "kube-api-access-mv4h7") pod "38c54c62-e5f5-4668-a43b-63d3760ef81b" (UID: "38c54c62-e5f5-4668-a43b-63d3760ef81b"). InnerVolumeSpecName "kube-api-access-mv4h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:51:08.310225 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.310165    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38c54c62-e5f5-4668-a43b-63d3760ef81b" (UID: "38c54c62-e5f5-4668-a43b-63d3760ef81b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:51:08.409548 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.409465    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:51:08.409548 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.409493    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv4h7\" (UniqueName: \"kubernetes.io/projected/38c54c62-e5f5-4668-a43b-63d3760ef81b-kube-api-access-mv4h7\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:51:08.409548 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.409503    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c54c62-e5f5-4668-a43b-63d3760ef81b-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:51:08.409548 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.409515    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c54c62-e5f5-4668-a43b-63d3760ef81b-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:51:08.997494 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.997463    2568 generic.go:356] "Generic (PLEG): container finished" podID="38c54c62-e5f5-4668-a43b-63d3760ef81b" containerID="6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f" exitCode=0
Sep 10 17:51:08.997903 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.997512    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"
Sep 10 17:51:08.997903 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.997547    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7" event={"ID":"38c54c62-e5f5-4668-a43b-63d3760ef81b","Type":"ContainerDied","Data":"6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f"}
Sep 10 17:51:08.997903 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.997604    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7" event={"ID":"38c54c62-e5f5-4668-a43b-63d3760ef81b","Type":"ContainerDied","Data":"3f166caab3eab928b28c01a221e75f81a0941974523fd0125d7fac131fa76e92"}
Sep 10 17:51:08.997903 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:08.997627    2568 scope.go:117] "RemoveContainer" containerID="6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f"
Sep 10 17:51:09.007528 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:09.007513    2568 scope.go:117] "RemoveContainer" containerID="6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f"
Sep 10 17:51:09.007785 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:51:09.007767    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f\": container with ID starting with 6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f not found: ID does not exist" containerID="6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f"
Sep 10 17:51:09.007826 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:09.007793    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f"} err="failed to get container status \"6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f\": rpc error: code = NotFound desc = could not find container \"6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f\": container with ID starting with 6732fbeabecb53894546d799b7cc5bb1dfb3873bcd8145e22c6832269efcb90f not found: ID does not exist"
Sep 10 17:51:09.023358 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:09.023334    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"]
Sep 10 17:51:09.026898 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:09.026876    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc47cd-696f8c546d-47cw7"]
Sep 10 17:51:09.769559 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:51:09.769525    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38c54c62-e5f5-4668-a43b-63d3760ef81b" path="/var/lib/kubelet/pods/38c54c62-e5f5-4668-a43b-63d3760ef81b/volumes"
Sep 10 17:52:39.020669 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.020635    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"]
Sep 10 17:52:39.021196 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.021130    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6" podUID="2ef0d88a-fd3a-4c65-9385-8f12495033b8" containerName="maas-api" containerID="cri-o://cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16" gracePeriod=30
Sep 10 17:52:39.084597 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.084561    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"]
Sep 10 17:52:39.084919 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.084860    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq" podUID="5ecf1b1b-854e-47d6-a199-5c4a516beb8c" containerName="payload-processing" containerID="cri-o://dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3" gracePeriod=30
Sep 10 17:52:39.090717 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.090679    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"]
Sep 10 17:52:39.090973 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.090913    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg" podUID="67d04d00-794f-47c6-9e0f-ef4a171d153a" containerName="payload-pre-processing" containerID="cri-o://889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f" gracePeriod=30
Sep 10 17:52:39.386184 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.386153    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:52:39.389044 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.389024    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:52:39.392331 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.392311    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:52:39.459932 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.459888    2568 generic.go:356] "Generic (PLEG): container finished" podID="5ecf1b1b-854e-47d6-a199-5c4a516beb8c" containerID="dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3" exitCode=0
Sep 10 17:52:39.459932 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.459935    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"
Sep 10 17:52:39.460201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.459995    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq" event={"ID":"5ecf1b1b-854e-47d6-a199-5c4a516beb8c","Type":"ContainerDied","Data":"dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3"}
Sep 10 17:52:39.460201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.460055    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq" event={"ID":"5ecf1b1b-854e-47d6-a199-5c4a516beb8c","Type":"ContainerDied","Data":"72bab4f5e4360befa64db58a526c1ce2ef3d8671def50e10e2688ed705812e49"}
Sep 10 17:52:39.460201 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.460084    2568 scope.go:117] "RemoveContainer" containerID="dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3"
Sep 10 17:52:39.461390 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.461371    2568 generic.go:356] "Generic (PLEG): container finished" podID="2ef0d88a-fd3a-4c65-9385-8f12495033b8" containerID="cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16" exitCode=0
Sep 10 17:52:39.461504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.461410    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"
Sep 10 17:52:39.461504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.461464    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6" event={"ID":"2ef0d88a-fd3a-4c65-9385-8f12495033b8","Type":"ContainerDied","Data":"cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16"}
Sep 10 17:52:39.461631 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.461516    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6" event={"ID":"2ef0d88a-fd3a-4c65-9385-8f12495033b8","Type":"ContainerDied","Data":"62feee123cdc4c80dedfa6af68929765df4a2b5d18762af50a18a52dbb3ca157"}
Sep 10 17:52:39.462866 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.462845    2568 generic.go:356] "Generic (PLEG): container finished" podID="67d04d00-794f-47c6-9e0f-ef4a171d153a" containerID="889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f" exitCode=0
Sep 10 17:52:39.462969 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.462929    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"
Sep 10 17:52:39.463013 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.462960    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg" event={"ID":"67d04d00-794f-47c6-9e0f-ef4a171d153a","Type":"ContainerDied","Data":"889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f"}
Sep 10 17:52:39.463050 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.463014    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg" event={"ID":"67d04d00-794f-47c6-9e0f-ef4a171d153a","Type":"ContainerDied","Data":"151a8391c09edcc0af352291d7ce1d795a30000ecb77087179c16ddb48c360dc"}
Sep 10 17:52:39.468921 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.468903    2568 scope.go:117] "RemoveContainer" containerID="dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3"
Sep 10 17:52:39.469205 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:52:39.469187    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3\": container with ID starting with dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3 not found: ID does not exist" containerID="dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3"
Sep 10 17:52:39.469272 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.469223    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3"} err="failed to get container status \"dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3\": rpc error: code = NotFound desc = could not find container \"dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3\": container with ID starting with dbdd6d8631b8672e86d1602aa4f0de1226bde66d4517e932f2bdcb5b0f2a04b3 not found: ID does not exist"
Sep 10 17:52:39.469272 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.469267    2568 scope.go:117] "RemoveContainer" containerID="cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16"
Sep 10 17:52:39.476991 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.476977    2568 scope.go:117] "RemoveContainer" containerID="cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16"
Sep 10 17:52:39.477262 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:52:39.477221    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16\": container with ID starting with cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16 not found: ID does not exist" containerID="cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16"
Sep 10 17:52:39.477262 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.477257    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16"} err="failed to get container status \"cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16\": rpc error: code = NotFound desc = could not find container \"cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16\": container with ID starting with cd0f02a955d802fae08e0ad564608d256428f5a6bc31e3ec693023a0ca762c16 not found: ID does not exist"
Sep 10 17:52:39.477419 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.477272    2568 scope.go:117] "RemoveContainer" containerID="889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f"
Sep 10 17:52:39.484344 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.484327    2568 scope.go:117] "RemoveContainer" containerID="889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f"
Sep 10 17:52:39.484574 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:52:39.484556    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f\": container with ID starting with 889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f not found: ID does not exist" containerID="889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f"
Sep 10 17:52:39.484620 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.484582    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f"} err="failed to get container status \"889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f\": rpc error: code = NotFound desc = could not find container \"889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f\": container with ID starting with 889be830e20da17c63f2ed2ee6853d951419365a1a2b6a2bc6804f664a9be99f not found: ID does not exist"
Sep 10 17:52:39.508869 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.508850    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls\") pod \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") "
Sep 10 17:52:39.508961 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.508879    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d04d00-794f-47c6-9e0f-ef4a171d153a-plugins-config-volume\") pod \"67d04d00-794f-47c6-9e0f-ef4a171d153a\" (UID: \"67d04d00-794f-47c6-9e0f-ef4a171d153a\") "
Sep 10 17:52:39.508961 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.508923    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-metrics-certs\") pod \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") "
Sep 10 17:52:39.509078 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509028    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p6gv\" (UniqueName: \"kubernetes.io/projected/2ef0d88a-fd3a-4c65-9385-8f12495033b8-kube-api-access-5p6gv\") pod \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") "
Sep 10 17:52:39.509078 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509061    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0d88a-fd3a-4c65-9385-8f12495033b8-openshift-service-ca\") pod \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\" (UID: \"2ef0d88a-fd3a-4c65-9385-8f12495033b8\") "
Sep 10 17:52:39.509192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509096    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq9c2\" (UniqueName: \"kubernetes.io/projected/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-kube-api-access-zq9c2\") pod \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\" (UID: \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\") "
Sep 10 17:52:39.509192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509120    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jzlx\" (UniqueName: \"kubernetes.io/projected/67d04d00-794f-47c6-9e0f-ef4a171d153a-kube-api-access-4jzlx\") pod \"67d04d00-794f-47c6-9e0f-ef4a171d153a\" (UID: \"67d04d00-794f-47c6-9e0f-ef4a171d153a\") "
Sep 10 17:52:39.509192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509151    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-plugins-config-volume\") pod \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\" (UID: \"5ecf1b1b-854e-47d6-a199-5c4a516beb8c\") "
Sep 10 17:52:39.509413 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509390    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d04d00-794f-47c6-9e0f-ef4a171d153a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67d04d00-794f-47c6-9e0f-ef4a171d153a" (UID: "67d04d00-794f-47c6-9e0f-ef4a171d153a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:39.509482 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509440    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef0d88a-fd3a-4c65-9385-8f12495033b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ef0d88a-fd3a-4c65-9385-8f12495033b8" (UID: "2ef0d88a-fd3a-4c65-9385-8f12495033b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:39.509482 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509466    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d04d00-794f-47c6-9e0f-ef4a171d153a-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:39.509743 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.509717    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ecf1b1b-854e-47d6-a199-5c4a516beb8c" (UID: "5ecf1b1b-854e-47d6-a199-5c4a516beb8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:39.511229 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.511203    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d04d00-794f-47c6-9e0f-ef4a171d153a-kube-api-access-4jzlx" (OuterVolumeSpecName: "kube-api-access-4jzlx") pod "67d04d00-794f-47c6-9e0f-ef4a171d153a" (UID: "67d04d00-794f-47c6-9e0f-ef4a171d153a"). InnerVolumeSpecName "kube-api-access-4jzlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:39.511311 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.511273    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-kube-api-access-zq9c2" (OuterVolumeSpecName: "kube-api-access-zq9c2") pod "5ecf1b1b-854e-47d6-a199-5c4a516beb8c" (UID: "5ecf1b1b-854e-47d6-a199-5c4a516beb8c"). InnerVolumeSpecName "kube-api-access-zq9c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:39.511311 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.511295    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ef0d88a-fd3a-4c65-9385-8f12495033b8" (UID: "2ef0d88a-fd3a-4c65-9385-8f12495033b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:39.511426 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.511378    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef0d88a-fd3a-4c65-9385-8f12495033b8-kube-api-access-5p6gv" (OuterVolumeSpecName: "kube-api-access-5p6gv") pod "2ef0d88a-fd3a-4c65-9385-8f12495033b8" (UID: "2ef0d88a-fd3a-4c65-9385-8f12495033b8"). InnerVolumeSpecName "kube-api-access-5p6gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:39.511426 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.511385    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ef0d88a-fd3a-4c65-9385-8f12495033b8" (UID: "2ef0d88a-fd3a-4c65-9385-8f12495033b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:39.610564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.610477    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p6gv\" (UniqueName: \"kubernetes.io/projected/2ef0d88a-fd3a-4c65-9385-8f12495033b8-kube-api-access-5p6gv\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:39.610564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.610506    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0d88a-fd3a-4c65-9385-8f12495033b8-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:39.610564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.610515    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq9c2\" (UniqueName: \"kubernetes.io/projected/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-kube-api-access-zq9c2\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:39.610564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.610524    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jzlx\" (UniqueName: \"kubernetes.io/projected/67d04d00-794f-47c6-9e0f-ef4a171d153a-kube-api-access-4jzlx\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:39.610564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.610533    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecf1b1b-854e-47d6-a199-5c4a516beb8c-plugins-config-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:39.610564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.610544    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:39.610564 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.610552    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef0d88a-fd3a-4c65-9385-8f12495033b8-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:39.788337 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.788311    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"]
Sep 10 17:52:39.791985 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.791965    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7621a1-67b69d4d98-kc7m6"]
Sep 10 17:52:39.803785 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.803765    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"]
Sep 10 17:52:39.807497 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.807478    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7621a1-bfd4597cd-h2wrg"]
Sep 10 17:52:39.818653 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.818635    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"]
Sep 10 17:52:39.820185 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:39.820167    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7621a1-5c7f44bd95-xfmdq"]
Sep 10 17:52:41.764281 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:41.764227    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ef0d88a-fd3a-4c65-9385-8f12495033b8" path="/var/lib/kubelet/pods/2ef0d88a-fd3a-4c65-9385-8f12495033b8/volumes"
Sep 10 17:52:41.764654 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:41.764600    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecf1b1b-854e-47d6-a199-5c4a516beb8c" path="/var/lib/kubelet/pods/5ecf1b1b-854e-47d6-a199-5c4a516beb8c/volumes"
Sep 10 17:52:41.764910 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:41.764897    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d04d00-794f-47c6-9e0f-ef4a171d153a" path="/var/lib/kubelet/pods/67d04d00-794f-47c6-9e0f-ef4a171d153a/volumes"
Sep 10 17:52:57.141079 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.141041    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"]
Sep 10 17:52:57.141511 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.141307    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2" podUID="5e57d2c2-7006-4c61-9bde-acde2cfe1e16" containerName="maas-api" containerID="cri-o://f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962" gracePeriod=30
Sep 10 17:52:57.390931 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.390908    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:52:57.555231 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.555197    2568 generic.go:356] "Generic (PLEG): container finished" podID="5e57d2c2-7006-4c61-9bde-acde2cfe1e16" containerID="f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962" exitCode=0
Sep 10 17:52:57.555404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.555254    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"
Sep 10 17:52:57.555404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.555280    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2" event={"ID":"5e57d2c2-7006-4c61-9bde-acde2cfe1e16","Type":"ContainerDied","Data":"f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962"}
Sep 10 17:52:57.555404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.555341    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2" event={"ID":"5e57d2c2-7006-4c61-9bde-acde2cfe1e16","Type":"ContainerDied","Data":"f4f0fb89eea4677d693e4018079495df6c33e5ff3fe388ecc3903f5f63d1fc2d"}
Sep 10 17:52:57.555404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.555371    2568 scope.go:117] "RemoveContainer" containerID="f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962"
Sep 10 17:52:57.564125 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.564108    2568 scope.go:117] "RemoveContainer" containerID="f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962"
Sep 10 17:52:57.564389 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:52:57.564371    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962\": container with ID starting with f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962 not found: ID does not exist" containerID="f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962"
Sep 10 17:52:57.564465 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.564399    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962"} err="failed to get container status \"f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962\": rpc error: code = NotFound desc = could not find container \"f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962\": container with ID starting with f51c26394d534efa46229cf7a9b70c91e646480ec34da1063dc0f737fc8b7962 not found: ID does not exist"
Sep 10 17:52:57.565631 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.565614    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls\") pod \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") "
Sep 10 17:52:57.565689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.565645    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-openshift-service-ca\") pod \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") "
Sep 10 17:52:57.565689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.565666    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-metrics-certs\") pod \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") "
Sep 10 17:52:57.565773 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.565740    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rczsb\" (UniqueName: \"kubernetes.io/projected/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-kube-api-access-rczsb\") pod \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\" (UID: \"5e57d2c2-7006-4c61-9bde-acde2cfe1e16\") "
Sep 10 17:52:57.565975 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.565952    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e57d2c2-7006-4c61-9bde-acde2cfe1e16" (UID: "5e57d2c2-7006-4c61-9bde-acde2cfe1e16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:52:57.567593 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.567569    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e57d2c2-7006-4c61-9bde-acde2cfe1e16" (UID: "5e57d2c2-7006-4c61-9bde-acde2cfe1e16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:57.567686 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.567675    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e57d2c2-7006-4c61-9bde-acde2cfe1e16" (UID: "5e57d2c2-7006-4c61-9bde-acde2cfe1e16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:52:57.567729 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.567688    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-kube-api-access-rczsb" (OuterVolumeSpecName: "kube-api-access-rczsb") pod "5e57d2c2-7006-4c61-9bde-acde2cfe1e16" (UID: "5e57d2c2-7006-4c61-9bde-acde2cfe1e16"). InnerVolumeSpecName "kube-api-access-rczsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:52:57.666875 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.666838    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczsb\" (UniqueName: \"kubernetes.io/projected/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-kube-api-access-rczsb\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:57.666875 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.666867    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:57.666875 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.666876    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:57.666875 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.666884    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57d2c2-7006-4c61-9bde-acde2cfe1e16-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:52:57.873054 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.872971    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"]
Sep 10 17:52:57.878956 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:57.878926    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-058069-68987cbc9d-5g8v2"]
Sep 10 17:52:59.763906 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:52:59.763872    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e57d2c2-7006-4c61-9bde-acde2cfe1e16" path="/var/lib/kubelet/pods/5e57d2c2-7006-4c61-9bde-acde2cfe1e16/volumes"
Sep 10 17:53:15.575283 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.575245    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"]
Sep 10 17:53:15.575902 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.575485    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56" podUID="ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" containerName="maas-api" containerID="cri-o://2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796" gracePeriod=30
Sep 10 17:53:15.825959 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.825902    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:53:15.910653 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.910614    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls\") pod \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") "
Sep 10 17:53:15.910860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.910666    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-metrics-certs\") pod \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") "
Sep 10 17:53:15.910860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.910723    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-openshift-service-ca\") pod \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") "
Sep 10 17:53:15.910860 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.910785    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wwnx\" (UniqueName: \"kubernetes.io/projected/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-kube-api-access-9wwnx\") pod \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\" (UID: \"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8\") "
Sep 10 17:53:15.911121 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.911096    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" (UID: "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:53:15.913003 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.912962    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-kube-api-access-9wwnx" (OuterVolumeSpecName: "kube-api-access-9wwnx") pod "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" (UID: "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8"). InnerVolumeSpecName "kube-api-access-9wwnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:53:15.913003 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.912975    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" (UID: "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:53:15.913192 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:15.913055    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" (UID: "ccfb3590-2d47-4e09-a2b3-2f34242aaaf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:53:16.012157 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.012116    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:53:16.012157 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.012150    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:53:16.012157 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.012160    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:53:16.012157 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.012169    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wwnx\" (UniqueName: \"kubernetes.io/projected/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8-kube-api-access-9wwnx\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:53:16.658748 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.658714    2568 generic.go:356] "Generic (PLEG): container finished" podID="ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" containerID="2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796" exitCode=0
Sep 10 17:53:16.659174 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.658780    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"
Sep 10 17:53:16.659174 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.658785    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56" event={"ID":"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8","Type":"ContainerDied","Data":"2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796"}
Sep 10 17:53:16.659174 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.658844    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56" event={"ID":"ccfb3590-2d47-4e09-a2b3-2f34242aaaf8","Type":"ContainerDied","Data":"9f6ad73b04c8e3d2b4e62f03338b3214bc412d7adc1669ce8e5d892839a3480e"}
Sep 10 17:53:16.659174 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.658868    2568 scope.go:117] "RemoveContainer" containerID="2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796"
Sep 10 17:53:16.668040 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.668020    2568 scope.go:117] "RemoveContainer" containerID="2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796"
Sep 10 17:53:16.668423 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:53:16.668393    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796\": container with ID starting with 2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796 not found: ID does not exist" containerID="2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796"
Sep 10 17:53:16.668531 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.668431    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796"} err="failed to get container status \"2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796\": rpc error: code = NotFound desc = could not find container \"2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796\": container with ID starting with 2c81ff5f3f448d1fc3312b84577aafc9fb5b8d5c53bcd6806514010337b9a796 not found: ID does not exist"
Sep 10 17:53:16.686534 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.686499    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"]
Sep 10 17:53:16.691871 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:53:16.691847    2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfb3590_2d47_4e09_a2b3_2f34242aaaf8.slice\": RecentStats: unable to find data in memory cache]"
Sep 10 17:53:16.694449 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:16.694426    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f14bf-78c7f5c7bb-2pf56"]
Sep 10 17:53:17.769115 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:17.769076    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" path="/var/lib/kubelet/pods/ccfb3590-2d47-4e09-a2b3-2f34242aaaf8/volumes"
Sep 10 17:53:25.337752 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.337715    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-t7s8h"]
Sep 10 17:53:25.338157 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.337982    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h" podUID="3a0c986c-7370-413b-b1ca-51ba670b1d65" containerName="manager" containerID="cri-o://7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c" gracePeriod=30
Sep 10 17:53:25.586003 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.585981    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:53:25.697541 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.697449    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmbwl\" (UniqueName: \"kubernetes.io/projected/3a0c986c-7370-413b-b1ca-51ba670b1d65-kube-api-access-mmbwl\") pod \"3a0c986c-7370-413b-b1ca-51ba670b1d65\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") "
Sep 10 17:53:25.697698 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.697601    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-metrics-certs\") pod \"3a0c986c-7370-413b-b1ca-51ba670b1d65\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") "
Sep 10 17:53:25.697698 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.697632    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-webhook-cert\") pod \"3a0c986c-7370-413b-b1ca-51ba670b1d65\" (UID: \"3a0c986c-7370-413b-b1ca-51ba670b1d65\") "
Sep 10 17:53:25.699628 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.699596    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3a0c986c-7370-413b-b1ca-51ba670b1d65" (UID: "3a0c986c-7370-413b-b1ca-51ba670b1d65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:53:25.699766 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.699687    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3a0c986c-7370-413b-b1ca-51ba670b1d65" (UID: "3a0c986c-7370-413b-b1ca-51ba670b1d65"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:53:25.699766 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.699726    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0c986c-7370-413b-b1ca-51ba670b1d65-kube-api-access-mmbwl" (OuterVolumeSpecName: "kube-api-access-mmbwl") pod "3a0c986c-7370-413b-b1ca-51ba670b1d65" (UID: "3a0c986c-7370-413b-b1ca-51ba670b1d65"). InnerVolumeSpecName "kube-api-access-mmbwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:53:25.705359 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.705186    2568 generic.go:356] "Generic (PLEG): container finished" podID="3a0c986c-7370-413b-b1ca-51ba670b1d65" containerID="7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c" exitCode=0
Sep 10 17:53:25.705359 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.705261    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h"
Sep 10 17:53:25.705359 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.705264    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h" event={"ID":"3a0c986c-7370-413b-b1ca-51ba670b1d65","Type":"ContainerDied","Data":"7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c"}
Sep 10 17:53:25.705359 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.705320    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-t7s8h" event={"ID":"3a0c986c-7370-413b-b1ca-51ba670b1d65","Type":"ContainerDied","Data":"3e4eb653072412095cd262d121215021fc12fd8eef57bff458e8b2979f3fd1e1"}
Sep 10 17:53:25.705359 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.705344    2568 scope.go:117] "RemoveContainer" containerID="7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c"
Sep 10 17:53:25.720702 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.720667    2568 scope.go:117] "RemoveContainer" containerID="7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c"
Sep 10 17:53:25.721039 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:53:25.721019    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c\": container with ID starting with 7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c not found: ID does not exist" containerID="7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c"
Sep 10 17:53:25.721111 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.721049    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c"} err="failed to get container status \"7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c\": rpc error: code = NotFound desc = could not find container \"7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c\": container with ID starting with 7e493add76e1b4ba432ca660a2b9db4f4ee559da40d7a4331f9ef4be14623b8c not found: ID does not exist"
Sep 10 17:53:25.733050 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.733025    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-t7s8h"]
Sep 10 17:53:25.737589 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.737569    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-t7s8h"]
Sep 10 17:53:25.764267 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.764219    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a0c986c-7370-413b-b1ca-51ba670b1d65" path="/var/lib/kubelet/pods/3a0c986c-7370-413b-b1ca-51ba670b1d65/volumes"
Sep 10 17:53:25.799306 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.799268    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:53:25.799306 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.799302    2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0c986c-7370-413b-b1ca-51ba670b1d65-webhook-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:53:25.799306 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:25.799311    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbwl\" (UniqueName: \"kubernetes.io/projected/3a0c986c-7370-413b-b1ca-51ba670b1d65-kube-api-access-mmbwl\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:53:51.842795 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:51.842767    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:53:51.846782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:53:51.846762    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:54:26.559970 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.559930    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-q5tpr"]
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560328    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560342    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560363    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecf1b1b-854e-47d6-a199-5c4a516beb8c" containerName="payload-processing"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560369    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecf1b1b-854e-47d6-a199-5c4a516beb8c" containerName="payload-processing"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560378    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb275291-d739-4585-a85f-c28436c69cf3" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560384    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb275291-d739-4585-a85f-c28436c69cf3" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560395    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e57d2c2-7006-4c61-9bde-acde2cfe1e16" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560401    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e57d2c2-7006-4c61-9bde-acde2cfe1e16" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560408    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d04d00-794f-47c6-9e0f-ef4a171d153a" containerName="payload-pre-processing"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560413    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d04d00-794f-47c6-9e0f-ef4a171d153a" containerName="payload-pre-processing"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560421    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38c54c62-e5f5-4668-a43b-63d3760ef81b" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560429    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="38c54c62-e5f5-4668-a43b-63d3760ef81b" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560445    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0c986c-7370-413b-b1ca-51ba670b1d65" containerName="manager"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560453    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0c986c-7370-413b-b1ca-51ba670b1d65" containerName="manager"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560461    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f24ee3-3cfc-424d-86ff-345627abd310" containerName="payload-processing"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560468    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f24ee3-3cfc-424d-86ff-345627abd310" containerName="payload-processing"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560479    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ef0d88a-fd3a-4c65-9385-8f12495033b8" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560487    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ef0d88a-fd3a-4c65-9385-8f12495033b8" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560571    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb275291-d739-4585-a85f-c28436c69cf3" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560583    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecf1b1b-854e-47d6-a199-5c4a516beb8c" containerName="payload-processing"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560593    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d04d00-794f-47c6-9e0f-ef4a171d153a" containerName="payload-pre-processing"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560605    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0c986c-7370-413b-b1ca-51ba670b1d65" containerName="manager"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560614    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfb3590-2d47-4e09-a2b3-2f34242aaaf8" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560621    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="38c54c62-e5f5-4668-a43b-63d3760ef81b" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560631    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e57d2c2-7006-4c61-9bde-acde2cfe1e16" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560641    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ef0d88a-fd3a-4c65-9385-8f12495033b8" containerName="maas-api"
Sep 10 17:54:26.560684 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.560650    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f24ee3-3cfc-424d-86ff-345627abd310" containerName="payload-processing"
Sep 10 17:54:26.563719 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.563703    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.567708 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.567685    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-spdc5\""
Sep 10 17:54:26.567708 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.567698    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 17:54:26.567872 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.567700    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 17:54:26.567872 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.567685    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 17:54:26.572685 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.572660    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-q5tpr"]
Sep 10 17:54:26.623301 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.623259    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-webhook-cert\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.623455 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.623332    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-metrics-certs\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.623455 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.623353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdc76\" (UniqueName: \"kubernetes.io/projected/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-kube-api-access-cdc76\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.724431 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.724391    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-webhook-cert\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.724620 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.724456    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-metrics-certs\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.724620 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.724486    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdc76\" (UniqueName: \"kubernetes.io/projected/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-kube-api-access-cdc76\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.727050 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.727025    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-metrics-certs\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.727137 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.727025    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-webhook-cert\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.733800 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.733770    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdc76\" (UniqueName: \"kubernetes.io/projected/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-kube-api-access-cdc76\") pod \"maas-controller-5c56cf8bbb-q5tpr\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") " pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:26.875721 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:26.875634    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:27.154965 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:27.154942    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-q5tpr"]
Sep 10 17:54:27.156153 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:54:27.156126    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c72c3f_8aa8_4a3f_b4ce_d6f599de6827.slice/crio-0940a65cca905d9d3a50a4424d001a22d9ee96e37eca4e23883909b620d4b44e WatchSource:0}: Error finding container 0940a65cca905d9d3a50a4424d001a22d9ee96e37eca4e23883909b620d4b44e: Status 404 returned error can't find the container with id 0940a65cca905d9d3a50a4424d001a22d9ee96e37eca4e23883909b620d4b44e
Sep 10 17:54:27.157361 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:27.157337    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:54:27.996330 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:27.996225    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr" event={"ID":"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827","Type":"ContainerStarted","Data":"6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6"}
Sep 10 17:54:27.996703 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:27.996331    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr" event={"ID":"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827","Type":"ContainerStarted","Data":"0940a65cca905d9d3a50a4424d001a22d9ee96e37eca4e23883909b620d4b44e"}
Sep 10 17:54:27.996703 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:27.996358    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:28.017096 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:28.017048    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr" podStartSLOduration=1.5762496430000001 podStartE2EDuration="2.01703391s" totalImagesPullingTime="440.784267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:54:28.015280269 +0000 UTC m=+1836.724874534" watchObservedRunningTime="2026-09-10 17:54:28.01703391 +0000 UTC m=+1836.726628179"
Sep 10 17:54:39.004665 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:39.004625    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:48.432767 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.432731    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-q5tpr"]
Sep 10 17:54:48.433186 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.432929    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr" podUID="68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" containerName="manager" containerID="cri-o://6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6" gracePeriod=30
Sep 10 17:54:48.681202 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.681175    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:48.824259 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.824204    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdc76\" (UniqueName: \"kubernetes.io/projected/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-kube-api-access-cdc76\") pod \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") "
Sep 10 17:54:48.824446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.824315    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-webhook-cert\") pod \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") "
Sep 10 17:54:48.824446 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.824367    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-metrics-certs\") pod \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\" (UID: \"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827\") "
Sep 10 17:54:48.826356 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.826314    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" (UID: "68c72c3f-8aa8-4a3f-b4ce-d6f599de6827"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:54:48.826499 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.826418    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" (UID: "68c72c3f-8aa8-4a3f-b4ce-d6f599de6827"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:54:48.826499 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.826477    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-kube-api-access-cdc76" (OuterVolumeSpecName: "kube-api-access-cdc76") pod "68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" (UID: "68c72c3f-8aa8-4a3f-b4ce-d6f599de6827"). InnerVolumeSpecName "kube-api-access-cdc76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:54:48.925086 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.925048    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:54:48.925086 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.925081    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdc76\" (UniqueName: \"kubernetes.io/projected/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-kube-api-access-cdc76\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:54:48.925329 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:48.925095    2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827-webhook-cert\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:54:49.098808 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.098714    2568 generic.go:356] "Generic (PLEG): container finished" podID="68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" containerID="6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6" exitCode=0
Sep 10 17:54:49.098808 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.098769    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr"
Sep 10 17:54:49.098999 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.098804    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr" event={"ID":"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827","Type":"ContainerDied","Data":"6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6"}
Sep 10 17:54:49.098999 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.098861    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-q5tpr" event={"ID":"68c72c3f-8aa8-4a3f-b4ce-d6f599de6827","Type":"ContainerDied","Data":"0940a65cca905d9d3a50a4424d001a22d9ee96e37eca4e23883909b620d4b44e"}
Sep 10 17:54:49.098999 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.098886    2568 scope.go:117] "RemoveContainer" containerID="6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6"
Sep 10 17:54:49.107734 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.107714    2568 scope.go:117] "RemoveContainer" containerID="6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6"
Sep 10 17:54:49.107980 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:54:49.107961    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6\": container with ID starting with 6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6 not found: ID does not exist" containerID="6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6"
Sep 10 17:54:49.108019 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.107993    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6"} err="failed to get container status \"6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6\": rpc error: code = NotFound desc = could not find container \"6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6\": container with ID starting with 6d4d4b8f9ec66c20dc9a335524a89110e2571bcd4333e5b8da767eebe061c5f6 not found: ID does not exist"
Sep 10 17:54:49.121371 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.121339    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-q5tpr"]
Sep 10 17:54:49.125725 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.125702    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-q5tpr"]
Sep 10 17:54:49.764285 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.764230    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" path="/var/lib/kubelet/pods/68c72c3f-8aa8-4a3f-b4ce-d6f599de6827/volumes"
Sep 10 17:54:49.962397 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.962365    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-b2x2j"]
Sep 10 17:54:49.962752 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.962739    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" containerName="manager"
Sep 10 17:54:49.962752 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.962753    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" containerName="manager"
Sep 10 17:54:49.962864 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.962817    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c72c3f-8aa8-4a3f-b4ce-d6f599de6827" containerName="manager"
Sep 10 17:54:49.967218 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.967185    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:49.971386 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.971362    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 17:54:49.971504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.971402    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 17:54:49.971504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.971402    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-spdc5\""
Sep 10 17:54:49.971504 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.971405    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 17:54:49.977155 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:49.977133    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-b2x2j"]
Sep 10 17:54:50.134848 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.134813    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79f67499-aa9a-4936-b8a4-ef419a5c065c-webhook-cert\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.135029 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.134862    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfplw\" (UniqueName: \"kubernetes.io/projected/79f67499-aa9a-4936-b8a4-ef419a5c065c-kube-api-access-vfplw\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.135029 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.134923    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f67499-aa9a-4936-b8a4-ef419a5c065c-metrics-certs\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.235692 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.235649    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f67499-aa9a-4936-b8a4-ef419a5c065c-metrics-certs\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.235900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.235721    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79f67499-aa9a-4936-b8a4-ef419a5c065c-webhook-cert\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.235900 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.235752    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfplw\" (UniqueName: \"kubernetes.io/projected/79f67499-aa9a-4936-b8a4-ef419a5c065c-kube-api-access-vfplw\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.238322 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.238297    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79f67499-aa9a-4936-b8a4-ef419a5c065c-webhook-cert\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.238431 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.238348    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f67499-aa9a-4936-b8a4-ef419a5c065c-metrics-certs\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.245140 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.245115    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfplw\" (UniqueName: \"kubernetes.io/projected/79f67499-aa9a-4936-b8a4-ef419a5c065c-kube-api-access-vfplw\") pod \"maas-controller-5c56cf8bbb-b2x2j\" (UID: \"79f67499-aa9a-4936-b8a4-ef419a5c065c\") " pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.278396 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.278362    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:50.553408 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:50.553374    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c56cf8bbb-b2x2j"]
Sep 10 17:54:50.554036 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:54:50.554009    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f67499_aa9a_4936_b8a4_ef419a5c065c.slice/crio-c303df31dd0d077b8376ba9b374144f3083501b40fd99df0f24b6c8641c18ae7 WatchSource:0}: Error finding container c303df31dd0d077b8376ba9b374144f3083501b40fd99df0f24b6c8641c18ae7: Status 404 returned error can't find the container with id c303df31dd0d077b8376ba9b374144f3083501b40fd99df0f24b6c8641c18ae7
Sep 10 17:54:51.111935 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:51.111898    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j" event={"ID":"79f67499-aa9a-4936-b8a4-ef419a5c065c","Type":"ContainerStarted","Data":"bf18226367607315efbcd8672e30a80ee447388f1bdafdc804fe9fab2fc044e6"}
Sep 10 17:54:51.112440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:51.111954    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j" event={"ID":"79f67499-aa9a-4936-b8a4-ef419a5c065c","Type":"ContainerStarted","Data":"c303df31dd0d077b8376ba9b374144f3083501b40fd99df0f24b6c8641c18ae7"}
Sep 10 17:54:51.112440 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:51.111997    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:54:51.132552 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:54:51.132504    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j" podStartSLOduration=1.7998911930000001 podStartE2EDuration="2.13248696s" totalImagesPullingTime="332.595767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:54:51.130857691 +0000 UTC m=+1859.840451960" watchObservedRunningTime="2026-09-10 17:54:51.13248696 +0000 UTC m=+1859.842081226"
Sep 10 17:55:02.118373 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:02.118339    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c56cf8bbb-b2x2j"
Sep 10 17:55:55.483450 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:55.483411    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"]
Sep 10 17:55:55.483958 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:55.483637    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj" podUID="3ccc5e0d-ad5b-4de7-b596-acb3e79809dc" containerName="manager" containerID="cri-o://6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd" gracePeriod=10
Sep 10 17:55:56.323133 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.323104    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:55:56.388595 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.388552    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-extensions-socket-volume\") pod \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\" (UID: \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\") "
Sep 10 17:55:56.388782 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.388635    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkhxg\" (UniqueName: \"kubernetes.io/projected/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-kube-api-access-dkhxg\") pod \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\" (UID: \"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc\") "
Sep 10 17:55:56.388943 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.388916    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3ccc5e0d-ad5b-4de7-b596-acb3e79809dc" (UID: "3ccc5e0d-ad5b-4de7-b596-acb3e79809dc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:55:56.390721 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.390699    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-kube-api-access-dkhxg" (OuterVolumeSpecName: "kube-api-access-dkhxg") pod "3ccc5e0d-ad5b-4de7-b596-acb3e79809dc" (UID: "3ccc5e0d-ad5b-4de7-b596-acb3e79809dc"). InnerVolumeSpecName "kube-api-access-dkhxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:55:56.431353 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.431317    2568 generic.go:356] "Generic (PLEG): container finished" podID="3ccc5e0d-ad5b-4de7-b596-acb3e79809dc" containerID="6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd" exitCode=0
Sep 10 17:55:56.431512 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.431378    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"
Sep 10 17:55:56.431512 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.431405    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj" event={"ID":"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc","Type":"ContainerDied","Data":"6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd"}
Sep 10 17:55:56.431512 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.431456    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj" event={"ID":"3ccc5e0d-ad5b-4de7-b596-acb3e79809dc","Type":"ContainerDied","Data":"64a9141bd0af7b9f88746a240b448b61826c6d1c114632f8d86c2b818391bf98"}
Sep 10 17:55:56.431512 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.431478    2568 scope.go:117] "RemoveContainer" containerID="6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd"
Sep 10 17:55:56.441029 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.441013    2568 scope.go:117] "RemoveContainer" containerID="6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd"
Sep 10 17:55:56.441324 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:55:56.441306    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd\": container with ID starting with 6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd not found: ID does not exist" containerID="6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd"
Sep 10 17:55:56.441380 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.441334    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd"} err="failed to get container status \"6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd\": rpc error: code = NotFound desc = could not find container \"6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd\": container with ID starting with 6ae8e33ec60b8bdeb3f6a97734fbafd753b37bed48f58b1153e9fee180810fbd not found: ID does not exist"
Sep 10 17:55:56.454899 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.454876    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"]
Sep 10 17:55:56.458264 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.458216    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rt9sj"]
Sep 10 17:55:56.489622 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.489590    2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-extensions-socket-volume\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:55:56.489622 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:56.489619    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkhxg\" (UniqueName: \"kubernetes.io/projected/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc-kube-api-access-dkhxg\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:55:57.763438 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:55:57.763409    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ccc5e0d-ad5b-4de7-b596-acb3e79809dc" path="/var/lib/kubelet/pods/3ccc5e0d-ad5b-4de7-b596-acb3e79809dc/volumes"
Sep 10 17:56:56.173463 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.173425    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"]
Sep 10 17:56:56.173908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.173787    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccc5e0d-ad5b-4de7-b596-acb3e79809dc" containerName="manager"
Sep 10 17:56:56.173908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.173798    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccc5e0d-ad5b-4de7-b596-acb3e79809dc" containerName="manager"
Sep 10 17:56:56.173908 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.173860    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccc5e0d-ad5b-4de7-b596-acb3e79809dc" containerName="manager"
Sep 10 17:56:56.176714 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.176698    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:56.179981 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.179956    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5qmxl\""
Sep 10 17:56:56.189924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.189892    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"]
Sep 10 17:56:56.287536 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.287493    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gzb7q\" (UID: \"b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:56.287536 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.287533    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhkx\" (UniqueName: \"kubernetes.io/projected/b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141-kube-api-access-5mhkx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gzb7q\" (UID: \"b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:56.388490 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.388453    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gzb7q\" (UID: \"b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:56.388490 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.388489    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhkx\" (UniqueName: \"kubernetes.io/projected/b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141-kube-api-access-5mhkx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gzb7q\" (UID: \"b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:56.388846 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.388823    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gzb7q\" (UID: \"b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:56.397560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.397527    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhkx\" (UniqueName: \"kubernetes.io/projected/b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141-kube-api-access-5mhkx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gzb7q\" (UID: \"b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:56.487062 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.486953    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:56.774131 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:56.774102    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"]
Sep 10 17:56:56.776153 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:56:56.776129    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b75e63_9a4e_475a_b9bf_bb0f8c5bc141.slice/crio-8e9c89038ba67f0c2a3fb921c1776f852e351c61a6797cddbdd05ca9ef68ab8a WatchSource:0}: Error finding container 8e9c89038ba67f0c2a3fb921c1776f852e351c61a6797cddbdd05ca9ef68ab8a: Status 404 returned error can't find the container with id 8e9c89038ba67f0c2a3fb921c1776f852e351c61a6797cddbdd05ca9ef68ab8a
Sep 10 17:56:57.717855 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:57.717812    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q" event={"ID":"b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141","Type":"ContainerStarted","Data":"8c66ef390a1d37eaec974bf90d4f5c72032aeb2e94ea3d3fbbb1ccea0adfee44"}
Sep 10 17:56:57.718267 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:57.717885    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:56:57.718267 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:57.717911    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q" event={"ID":"b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141","Type":"ContainerStarted","Data":"8e9c89038ba67f0c2a3fb921c1776f852e351c61a6797cddbdd05ca9ef68ab8a"}
Sep 10 17:56:57.739276 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:56:57.739192    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q" podStartSLOduration=1.73917826 podStartE2EDuration="1.73917826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:56:57.736902834 +0000 UTC m=+1986.446497099" watchObservedRunningTime="2026-09-10 17:56:57.73917826 +0000 UTC m=+1986.448772526"
Sep 10 17:57:08.725255 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:08.725212    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gzb7q"
Sep 10 17:57:40.903828 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.903794    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"]
Sep 10 17:57:40.907452 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.907433    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:40.911222 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.911200    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-21b56a28-kserve-self-signed-certs\""
Sep 10 17:57:40.918417 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.918395    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"]
Sep 10 17:57:40.967067 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.967027    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-dshm\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:40.967067 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.967067    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-home\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:40.967288 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.967086    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tls-certs\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:40.967288 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.967144    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-model-cache\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:40.967288 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.967205    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvzf\" (UniqueName: \"kubernetes.io/projected/57996fdd-fe78-4cbc-b691-c57e3f0acd72-kube-api-access-5jvzf\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:40.967395 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:40.967292    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tmp-dir\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068394 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068362    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-dshm\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068394 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068396    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-home\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068618 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068416    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tls-certs\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068618 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068451    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-model-cache\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068618 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068487    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvzf\" (UniqueName: \"kubernetes.io/projected/57996fdd-fe78-4cbc-b691-c57e3f0acd72-kube-api-access-5jvzf\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068618 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068537    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tmp-dir\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068845 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068823    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-home\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068926 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068902    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-model-cache\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.068982 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.068933    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tmp-dir\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.070733 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.070707    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-dshm\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.071136 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.071117    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tls-certs\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.077203 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.077173    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvzf\" (UniqueName: \"kubernetes.io/projected/57996fdd-fe78-4cbc-b691-c57e3f0acd72-kube-api-access-5jvzf\") pod \"e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") " pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.218489 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.218399    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:41.491164 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.491140    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"]
Sep 10 17:57:41.493152 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:57:41.493125    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57996fdd_fe78_4cbc_b691_c57e3f0acd72.slice/crio-c823406af80c37f308609bd44676fa7e18078e9adc1f1e75141544f46bad9ebf WatchSource:0}: Error finding container c823406af80c37f308609bd44676fa7e18078e9adc1f1e75141544f46bad9ebf: Status 404 returned error can't find the container with id c823406af80c37f308609bd44676fa7e18078e9adc1f1e75141544f46bad9ebf
Sep 10 17:57:41.934477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.934438    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r" event={"ID":"57996fdd-fe78-4cbc-b691-c57e3f0acd72","Type":"ContainerStarted","Data":"698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38"}
Sep 10 17:57:41.934477 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.934481    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r" event={"ID":"57996fdd-fe78-4cbc-b691-c57e3f0acd72","Type":"ContainerStarted","Data":"c823406af80c37f308609bd44676fa7e18078e9adc1f1e75141544f46bad9ebf"}
Sep 10 17:57:41.957868 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:41.957809    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r" podStartSLOduration=1.957790288 podStartE2EDuration="1.957790288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:57:41.955661014 +0000 UTC m=+2030.665255279" watchObservedRunningTime="2026-09-10 17:57:41.957790288 +0000 UTC m=+2030.667384553"
Sep 10 17:57:46.597171 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.597129    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"]
Sep 10 17:57:46.600776 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.600755    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.603697 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.603674    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-21b56a28-kserve-self-signed-certs\""
Sep 10 17:57:46.612771 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.612746    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"]
Sep 10 17:57:46.718400 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.718361    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfpv2\" (UniqueName: \"kubernetes.io/projected/da5af8b7-8a03-43fe-941d-04329127ed6f-kube-api-access-dfpv2\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.718400 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.718409    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-home\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.718619 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.718430    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5af8b7-8a03-43fe-941d-04329127ed6f-tls-certs\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.718619 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.718547    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-tmp-dir\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.718619 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.718586    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-dshm\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.718720 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.718676    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-model-cache\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.819891 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.819855    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-model-cache\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.819891 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.819897    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfpv2\" (UniqueName: \"kubernetes.io/projected/da5af8b7-8a03-43fe-941d-04329127ed6f-kube-api-access-dfpv2\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.820133 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.819920    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-home\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.820133 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.819939    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5af8b7-8a03-43fe-941d-04329127ed6f-tls-certs\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.820133 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.819985    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-tmp-dir\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.820133 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.820007    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-dshm\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.820395 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.820369    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-model-cache\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.820460 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.820368    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-home\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.820460 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.820442    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-tmp-dir\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.822305 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.822275    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-dshm\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.822501 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.822485    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5af8b7-8a03-43fe-941d-04329127ed6f-tls-certs\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.827701 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.827676    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfpv2\" (UniqueName: \"kubernetes.io/projected/da5af8b7-8a03-43fe-941d-04329127ed6f-kube-api-access-dfpv2\") pod \"e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") " pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:46.913046 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:46.912984    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:47.192537 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:47.192514    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"]
Sep 10 17:57:47.194541 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:57:47.194515    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda5af8b7_8a03_43fe_941d_04329127ed6f.slice/crio-cbeb9675b337ec2125f8cec925dc7be71e14a66cfcf1631f5f7481632a3ff1f5 WatchSource:0}: Error finding container cbeb9675b337ec2125f8cec925dc7be71e14a66cfcf1631f5f7481632a3ff1f5: Status 404 returned error can't find the container with id cbeb9675b337ec2125f8cec925dc7be71e14a66cfcf1631f5f7481632a3ff1f5
Sep 10 17:57:47.966536 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:47.966496    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp" event={"ID":"da5af8b7-8a03-43fe-941d-04329127ed6f","Type":"ContainerStarted","Data":"4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686"}
Sep 10 17:57:47.966913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:47.966544    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp" event={"ID":"da5af8b7-8a03-43fe-941d-04329127ed6f","Type":"ContainerStarted","Data":"cbeb9675b337ec2125f8cec925dc7be71e14a66cfcf1631f5f7481632a3ff1f5"}
Sep 10 17:57:47.985432 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:47.985374    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp" podStartSLOduration=1.985357992 podStartE2EDuration="1.985357992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:57:47.983218671 +0000 UTC m=+2036.692812937" watchObservedRunningTime="2026-09-10 17:57:47.985357992 +0000 UTC m=+2036.694952257"
Sep 10 17:57:51.218941 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:51.218882    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:51.219376 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:51.218959    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:51.231509 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:51.231479    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:51.996086 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:51.996055    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:57:56.913822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:56.913787    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:56.914228 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:56.913835    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:56.926292 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:56.926254    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:57:57.021340 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:57:57.021311    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:58:42.249774 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:42.249729    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"]
Sep 10 17:58:43.218544 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:43.218509    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"]
Sep 10 17:58:51.876585 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:51.876557    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:58:51.883103 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:51.883081    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-znlbh_f7bc1e14-5460-41d2-a808-8ebb92f3c29f/ovn-acl-logging/0.log"
Sep 10 17:58:57.274441 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.274391    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r" podUID="57996fdd-fe78-4cbc-b691-c57e3f0acd72" containerName="main" containerID="cri-o://698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38" gracePeriod=45
Sep 10 17:58:57.520433 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.520410    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:58:57.542165 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542080    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tls-certs\") pod \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") "
Sep 10 17:58:57.542165 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542128    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tmp-dir\") pod \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") "
Sep 10 17:58:57.542404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542169    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jvzf\" (UniqueName: \"kubernetes.io/projected/57996fdd-fe78-4cbc-b691-c57e3f0acd72-kube-api-access-5jvzf\") pod \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") "
Sep 10 17:58:57.542404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542230    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-model-cache\") pod \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") "
Sep 10 17:58:57.542404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542300    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-dshm\") pod \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") "
Sep 10 17:58:57.542404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542366    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-home\") pod \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\" (UID: \"57996fdd-fe78-4cbc-b691-c57e3f0acd72\") "
Sep 10 17:58:57.542913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542761    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-home" (OuterVolumeSpecName: "home") pod "57996fdd-fe78-4cbc-b691-c57e3f0acd72" (UID: "57996fdd-fe78-4cbc-b691-c57e3f0acd72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:58:57.542913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542782    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-model-cache" (OuterVolumeSpecName: "model-cache") pod "57996fdd-fe78-4cbc-b691-c57e3f0acd72" (UID: "57996fdd-fe78-4cbc-b691-c57e3f0acd72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:58:57.543079 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.542930    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57996fdd-fe78-4cbc-b691-c57e3f0acd72" (UID: "57996fdd-fe78-4cbc-b691-c57e3f0acd72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:58:57.545462 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.545436    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-dshm" (OuterVolumeSpecName: "dshm") pod "57996fdd-fe78-4cbc-b691-c57e3f0acd72" (UID: "57996fdd-fe78-4cbc-b691-c57e3f0acd72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:58:57.545462 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.545436    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57996fdd-fe78-4cbc-b691-c57e3f0acd72" (UID: "57996fdd-fe78-4cbc-b691-c57e3f0acd72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:58:57.545628 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.545461    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57996fdd-fe78-4cbc-b691-c57e3f0acd72-kube-api-access-5jvzf" (OuterVolumeSpecName: "kube-api-access-5jvzf") pod "57996fdd-fe78-4cbc-b691-c57e3f0acd72" (UID: "57996fdd-fe78-4cbc-b691-c57e3f0acd72"). InnerVolumeSpecName "kube-api-access-5jvzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:58:57.643685 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.643650    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:57.643685 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.643678    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:57.643994 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.643705    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:57.643994 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.643716    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvzf\" (UniqueName: \"kubernetes.io/projected/57996fdd-fe78-4cbc-b691-c57e3f0acd72-kube-api-access-5jvzf\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:57.643994 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.643726    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:57.643994 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:57.643733    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57996fdd-fe78-4cbc-b691-c57e3f0acd72-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:58.239169 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.239110    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp" podUID="da5af8b7-8a03-43fe-941d-04329127ed6f" containerName="main" containerID="cri-o://4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686" gracePeriod=45
Sep 10 17:58:58.320962 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.320925    2568 generic.go:356] "Generic (PLEG): container finished" podID="57996fdd-fe78-4cbc-b691-c57e3f0acd72" containerID="698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38" exitCode=0
Sep 10 17:58:58.321361 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.320991    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"
Sep 10 17:58:58.321361 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.321014    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r" event={"ID":"57996fdd-fe78-4cbc-b691-c57e3f0acd72","Type":"ContainerDied","Data":"698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38"}
Sep 10 17:58:58.321361 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.321062    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r" event={"ID":"57996fdd-fe78-4cbc-b691-c57e3f0acd72","Type":"ContainerDied","Data":"c823406af80c37f308609bd44676fa7e18078e9adc1f1e75141544f46bad9ebf"}
Sep 10 17:58:58.321361 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.321079    2568 scope.go:117] "RemoveContainer" containerID="698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38"
Sep 10 17:58:58.339112 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.339083    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"]
Sep 10 17:58:58.344006 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.343969    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-21b56a28-kserve-9984dfbcf-6x87r"]
Sep 10 17:58:58.350740 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.350718    2568 scope.go:117] "RemoveContainer" containerID="698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38"
Sep 10 17:58:58.351132 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:58:58.351102    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38\": container with ID starting with 698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38 not found: ID does not exist" containerID="698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38"
Sep 10 17:58:58.351211 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.351145    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38"} err="failed to get container status \"698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38\": rpc error: code = NotFound desc = could not find container \"698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38\": container with ID starting with 698c42f5fddd05a9933dacad7e04bf60a997d64bb5246b93e4279b65ba2b8e38 not found: ID does not exist"
Sep 10 17:58:58.485404 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.485378    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:58:58.553294 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553265    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-home\") pod \"da5af8b7-8a03-43fe-941d-04329127ed6f\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") "
Sep 10 17:58:58.553294 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553306    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-model-cache\") pod \"da5af8b7-8a03-43fe-941d-04329127ed6f\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") "
Sep 10 17:58:58.553532 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553329    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfpv2\" (UniqueName: \"kubernetes.io/projected/da5af8b7-8a03-43fe-941d-04329127ed6f-kube-api-access-dfpv2\") pod \"da5af8b7-8a03-43fe-941d-04329127ed6f\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") "
Sep 10 17:58:58.553532 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553430    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5af8b7-8a03-43fe-941d-04329127ed6f-tls-certs\") pod \"da5af8b7-8a03-43fe-941d-04329127ed6f\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") "
Sep 10 17:58:58.553532 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553456    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-tmp-dir\") pod \"da5af8b7-8a03-43fe-941d-04329127ed6f\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") "
Sep 10 17:58:58.553532 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553481    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-dshm\") pod \"da5af8b7-8a03-43fe-941d-04329127ed6f\" (UID: \"da5af8b7-8a03-43fe-941d-04329127ed6f\") "
Sep 10 17:58:58.553745 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553544    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "da5af8b7-8a03-43fe-941d-04329127ed6f" (UID: "da5af8b7-8a03-43fe-941d-04329127ed6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:58:58.553745 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553576    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-home" (OuterVolumeSpecName: "home") pod "da5af8b7-8a03-43fe-941d-04329127ed6f" (UID: "da5af8b7-8a03-43fe-941d-04329127ed6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:58:58.553745 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553728    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da5af8b7-8a03-43fe-941d-04329127ed6f" (UID: "da5af8b7-8a03-43fe-941d-04329127ed6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:58:58.553901 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553737    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-home\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:58.553901 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.553778    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-model-cache\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:58.555685 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.555658    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5af8b7-8a03-43fe-941d-04329127ed6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da5af8b7-8a03-43fe-941d-04329127ed6f" (UID: "da5af8b7-8a03-43fe-941d-04329127ed6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:58:58.555791 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.555736    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-dshm" (OuterVolumeSpecName: "dshm") pod "da5af8b7-8a03-43fe-941d-04329127ed6f" (UID: "da5af8b7-8a03-43fe-941d-04329127ed6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:58:58.555945 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.555922    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5af8b7-8a03-43fe-941d-04329127ed6f-kube-api-access-dfpv2" (OuterVolumeSpecName: "kube-api-access-dfpv2") pod "da5af8b7-8a03-43fe-941d-04329127ed6f" (UID: "da5af8b7-8a03-43fe-941d-04329127ed6f"). InnerVolumeSpecName "kube-api-access-dfpv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:58:58.654920 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.654884    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5af8b7-8a03-43fe-941d-04329127ed6f-tls-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:58.654920 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.654911    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-tmp-dir\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:58.654920 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.654919    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da5af8b7-8a03-43fe-941d-04329127ed6f-dshm\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:58.654920 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:58.654928    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfpv2\" (UniqueName: \"kubernetes.io/projected/da5af8b7-8a03-43fe-941d-04329127ed6f-kube-api-access-dfpv2\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:58:59.328692 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.328653    2568 generic.go:356] "Generic (PLEG): container finished" podID="da5af8b7-8a03-43fe-941d-04329127ed6f" containerID="4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686" exitCode=0
Sep 10 17:58:59.329145 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.328698    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"
Sep 10 17:58:59.329145 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.328721    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp" event={"ID":"da5af8b7-8a03-43fe-941d-04329127ed6f","Type":"ContainerDied","Data":"4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686"}
Sep 10 17:58:59.329145 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.328762    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp" event={"ID":"da5af8b7-8a03-43fe-941d-04329127ed6f","Type":"ContainerDied","Data":"cbeb9675b337ec2125f8cec925dc7be71e14a66cfcf1631f5f7481632a3ff1f5"}
Sep 10 17:58:59.329145 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.328779    2568 scope.go:117] "RemoveContainer" containerID="4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686"
Sep 10 17:58:59.337910 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.337893    2568 scope.go:117] "RemoveContainer" containerID="4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686"
Sep 10 17:58:59.338196 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:58:59.338179    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686\": container with ID starting with 4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686 not found: ID does not exist" containerID="4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686"
Sep 10 17:58:59.338269 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.338202    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686"} err="failed to get container status \"4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686\": rpc error: code = NotFound desc = could not find container \"4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686\": container with ID starting with 4ab49011d4706c094d5a3f91031613ea986d4398a0af3ba4c5e6c565bb097686 not found: ID does not exist"
Sep 10 17:58:59.351675 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.351643    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"]
Sep 10 17:58:59.354207 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.354183    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-21b56a28-kserve-567dfc4499-rkjmp"]
Sep 10 17:58:59.764260 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.764168    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57996fdd-fe78-4cbc-b691-c57e3f0acd72" path="/var/lib/kubelet/pods/57996fdd-fe78-4cbc-b691-c57e3f0acd72/volumes"
Sep 10 17:58:59.764675 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:58:59.764657    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da5af8b7-8a03-43fe-941d-04329127ed6f" path="/var/lib/kubelet/pods/da5af8b7-8a03-43fe-941d-04329127ed6f/volumes"
Sep 10 17:59:01.221206 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.221163    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"]
Sep 10 17:59:01.221766 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.221744    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57996fdd-fe78-4cbc-b691-c57e3f0acd72" containerName="main"
Sep 10 17:59:01.221828 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.221771    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="57996fdd-fe78-4cbc-b691-c57e3f0acd72" containerName="main"
Sep 10 17:59:01.221828 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.221799    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5af8b7-8a03-43fe-941d-04329127ed6f" containerName="main"
Sep 10 17:59:01.221828 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.221809    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5af8b7-8a03-43fe-941d-04329127ed6f" containerName="main"
Sep 10 17:59:01.221944 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.221922    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5af8b7-8a03-43fe-941d-04329127ed6f" containerName="main"
Sep 10 17:59:01.221944 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.221936    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="57996fdd-fe78-4cbc-b691-c57e3f0acd72" containerName="main"
Sep 10 17:59:01.226689 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.226667    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.237784 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.237754    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"]
Sep 10 17:59:01.276287 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.276253    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dcfae0-3882-470a-a377-e196f18182b1-metrics-certs\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.276435 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.276351    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96dcfae0-3882-470a-a377-e196f18182b1-openshift-service-ca\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.276435 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.276382    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdlf\" (UniqueName: \"kubernetes.io/projected/96dcfae0-3882-470a-a377-e196f18182b1-kube-api-access-5vdlf\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.276435 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.276408    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96dcfae0-3882-470a-a377-e196f18182b1-maas-api-tls\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.377059 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.377023    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dcfae0-3882-470a-a377-e196f18182b1-metrics-certs\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.377289 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.377079    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96dcfae0-3882-470a-a377-e196f18182b1-openshift-service-ca\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.377289 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.377098    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdlf\" (UniqueName: \"kubernetes.io/projected/96dcfae0-3882-470a-a377-e196f18182b1-kube-api-access-5vdlf\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.377289 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.377121    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96dcfae0-3882-470a-a377-e196f18182b1-maas-api-tls\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.377734 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.377710    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96dcfae0-3882-470a-a377-e196f18182b1-openshift-service-ca\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.379558 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.379535    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96dcfae0-3882-470a-a377-e196f18182b1-maas-api-tls\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.379670 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.379624    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dcfae0-3882-470a-a377-e196f18182b1-metrics-certs\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.384759 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.384741    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdlf\" (UniqueName: \"kubernetes.io/projected/96dcfae0-3882-470a-a377-e196f18182b1-kube-api-access-5vdlf\") pod \"maas-api-c94d45875-r5x9r\" (UID: \"96dcfae0-3882-470a-a377-e196f18182b1\") " pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.538391 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.538360    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:01.808298 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:01.808274    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"]
Sep 10 17:59:01.810659 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:59:01.810635    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96dcfae0_3882_470a_a377_e196f18182b1.slice/crio-3506a2076eb25f572b21ad5fa792c32a7ef3bf9d71457bdc8c293ed8b0ec059d WatchSource:0}: Error finding container 3506a2076eb25f572b21ad5fa792c32a7ef3bf9d71457bdc8c293ed8b0ec059d: Status 404 returned error can't find the container with id 3506a2076eb25f572b21ad5fa792c32a7ef3bf9d71457bdc8c293ed8b0ec059d
Sep 10 17:59:02.347760 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:02.347723    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r" event={"ID":"96dcfae0-3882-470a-a377-e196f18182b1","Type":"ContainerStarted","Data":"5c74471ce752c45c6b64903a40607aef94502e371b53816d030db52c3baffa29"}
Sep 10 17:59:02.348274 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:02.347788    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r" event={"ID":"96dcfae0-3882-470a-a377-e196f18182b1","Type":"ContainerStarted","Data":"3506a2076eb25f572b21ad5fa792c32a7ef3bf9d71457bdc8c293ed8b0ec059d"}
Sep 10 17:59:02.348274 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:02.347852    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:02.373257 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:02.370523    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r" podStartSLOduration=0.968295332 podStartE2EDuration="1.370502903s" totalImagesPullingTime="402.207571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:59:02.368208747 +0000 UTC m=+2111.077803016" watchObservedRunningTime="2026-09-10 17:59:02.370502903 +0000 UTC m=+2111.080097171"
Sep 10 17:59:08.357823 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:08.357791    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c94d45875-r5x9r"
Sep 10 17:59:15.829894 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:15.829844    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"]
Sep 10 17:59:15.830348 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:15.830055    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h" podUID="85030212-3063-4745-aa1b-4311abd98b13" containerName="maas-api" containerID="cri-o://ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4" gracePeriod=30
Sep 10 17:59:17.080268 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.080225    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:59:17.215336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.215220    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-metrics-certs\") pod \"85030212-3063-4745-aa1b-4311abd98b13\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") "
Sep 10 17:59:17.215336 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.215292    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq7p8\" (UniqueName: \"kubernetes.io/projected/85030212-3063-4745-aa1b-4311abd98b13-kube-api-access-bq7p8\") pod \"85030212-3063-4745-aa1b-4311abd98b13\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") "
Sep 10 17:59:17.215582 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.215356    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-maas-api-tls\") pod \"85030212-3063-4745-aa1b-4311abd98b13\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") "
Sep 10 17:59:17.215582 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.215418    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85030212-3063-4745-aa1b-4311abd98b13-openshift-service-ca\") pod \"85030212-3063-4745-aa1b-4311abd98b13\" (UID: \"85030212-3063-4745-aa1b-4311abd98b13\") "
Sep 10 17:59:17.215873 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.215845    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85030212-3063-4745-aa1b-4311abd98b13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85030212-3063-4745-aa1b-4311abd98b13" (UID: "85030212-3063-4745-aa1b-4311abd98b13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:59:17.217449 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.217429    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "85030212-3063-4745-aa1b-4311abd98b13" (UID: "85030212-3063-4745-aa1b-4311abd98b13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:59:17.217538 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.217493    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85030212-3063-4745-aa1b-4311abd98b13" (UID: "85030212-3063-4745-aa1b-4311abd98b13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:59:17.217579 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.217564    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85030212-3063-4745-aa1b-4311abd98b13-kube-api-access-bq7p8" (OuterVolumeSpecName: "kube-api-access-bq7p8") pod "85030212-3063-4745-aa1b-4311abd98b13" (UID: "85030212-3063-4745-aa1b-4311abd98b13"). InnerVolumeSpecName "kube-api-access-bq7p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:59:17.316737 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.316697    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-maas-api-tls\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:59:17.316737 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.316734    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85030212-3063-4745-aa1b-4311abd98b13-openshift-service-ca\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:59:17.316737 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.316743    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85030212-3063-4745-aa1b-4311abd98b13-metrics-certs\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:59:17.316985 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.316752    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq7p8\" (UniqueName: \"kubernetes.io/projected/85030212-3063-4745-aa1b-4311abd98b13-kube-api-access-bq7p8\") on node \"ip-10-0-142-242.ec2.internal\" DevicePath \"\""
Sep 10 17:59:17.421323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.421279    2568 generic.go:356] "Generic (PLEG): container finished" podID="85030212-3063-4745-aa1b-4311abd98b13" containerID="ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4" exitCode=0
Sep 10 17:59:17.421323 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.421327    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"
Sep 10 17:59:17.421527 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.421366    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h" event={"ID":"85030212-3063-4745-aa1b-4311abd98b13","Type":"ContainerDied","Data":"ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4"}
Sep 10 17:59:17.421527 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.421413    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h" event={"ID":"85030212-3063-4745-aa1b-4311abd98b13","Type":"ContainerDied","Data":"3c794fb07dc6bc8d395dd8fab5c4d759a6e4b78b8ec210f11f426669bb2153dc"}
Sep 10 17:59:17.421527 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.421435    2568 scope.go:117] "RemoveContainer" containerID="ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4"
Sep 10 17:59:17.431071 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.431054    2568 scope.go:117] "RemoveContainer" containerID="ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4"
Sep 10 17:59:17.431358 ip-10-0-142-242 kubenswrapper[2568]: E0910 17:59:17.431337    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4\": container with ID starting with ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4 not found: ID does not exist" containerID="ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4"
Sep 10 17:59:17.431413 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.431367    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4"} err="failed to get container status \"ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4\": rpc error: code = NotFound desc = could not find container \"ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4\": container with ID starting with ad31e5e64b6374895de9d5ab3774ea1033927f6a53df0b148c08a21978a0e1b4 not found: ID does not exist"
Sep 10 17:59:17.443661 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.443631    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"]
Sep 10 17:59:17.449342 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.449319    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84847f94bc-4gg6h"]
Sep 10 17:59:17.763490 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:17.763454    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85030212-3063-4745-aa1b-4311abd98b13" path="/var/lib/kubelet/pods/85030212-3063-4745-aa1b-4311abd98b13/volumes"
Sep 10 17:59:37.022825 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:37.022792    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lrltm_ac39f1ad-5fd6-4552-93c9-ffd44795cd5a/manager/0.log"
Sep 10 17:59:37.139667 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:37.139633    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4zr7p_2e7225d6-da90-4e22-88ba-d5bb2556cf79/manager/0.log"
Sep 10 17:59:37.273854 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:37.273493    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c56cf8bbb-b2x2j_79f67499-aa9a-4936-b8a4-ef419a5c065c/manager/0.log"
Sep 10 17:59:37.393184 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:37.393144    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9z5gj_b59c7cdb-b7b3-4acb-b907-76dee58922f2/server/0.log"
Sep 10 17:59:37.881858 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:37.881829    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b6bb7766-xfgqm_763ecc42-f2ff-48ff-ac06-20fa3c0b0856/copy-manifests/0.log"
Sep 10 17:59:37.897911 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:37.897882    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b6bb7766-xfgqm_763ecc42-f2ff-48ff-ac06-20fa3c0b0856/manager/0.log"
Sep 10 17:59:38.360771 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:38.360734    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c94d45875-r5x9r_96dcfae0-3882-470a-a377-e196f18182b1/maas-api/0.log"
Sep 10 17:59:38.581683 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:38.581653    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5d2zw_7f00e367-c983-489a-a34c-f02cd6957354/postgres/0.log"
Sep 10 17:59:39.508581 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:39.508539    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-887dw_b0a3af90-b4f8-4610-8c85-c2e6603d1a67/manager/0.log"
Sep 10 17:59:39.630187 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:39.630156    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-6qqc6_9acead52-cfd6-48ae-9d7e-89cc1e472a93/manager/0.log"
Sep 10 17:59:39.858472 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:39.858428    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gzb7q_b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141/manager/0.log"
Sep 10 17:59:40.099420 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:40.099377    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-mtmz2_e357f2b7-993e-433c-bc09-5d2e65445339/manager/0.log"
Sep 10 17:59:40.792677 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:40.792647    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf54bbc88-h467l_1654d45e-f4d6-4a2f-9c4c-f72633158a71/kube-auth-proxy/0.log"
Sep 10 17:59:41.038924 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:41.038895    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-s27sr_008bb171-a493-4df9-b15e-e2d8a23ad60e/payload-pre-processing/0.log"
Sep 10 17:59:41.845138 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:41.845093    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-4sjxm_0b289abc-e9fd-40f7-aa71-d008dbb47072/main/0.log"
Sep 10 17:59:41.952492 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:41.952464    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5xfzp_05f50b85-67cc-462c-a477-dfa9e58948b7/main/0.log"
Sep 10 17:59:42.065688 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:42.065660    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frfwpq_ea8335d9-ac37-4fe9-9909-782e118c2a22/main/0.log"
Sep 10 17:59:47.896560 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.896476    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8jqk/must-gather-dbncz"]
Sep 10 17:59:47.896937 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.896887    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85030212-3063-4745-aa1b-4311abd98b13" containerName="maas-api"
Sep 10 17:59:47.896937 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.896900    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85030212-3063-4745-aa1b-4311abd98b13" containerName="maas-api"
Sep 10 17:59:47.897017 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.896953    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="85030212-3063-4745-aa1b-4311abd98b13" containerName="maas-api"
Sep 10 17:59:47.900205 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.900184    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8jqk/must-gather-dbncz"
Sep 10 17:59:47.904740 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.904718    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8jqk\"/\"kube-root-ca.crt\""
Sep 10 17:59:47.904894 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.904874    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8jqk\"/\"openshift-service-ca.crt\""
Sep 10 17:59:47.905858 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.905842    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8jqk\"/\"default-dockercfg-hcms2\""
Sep 10 17:59:47.911132 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.911113    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8jqk/must-gather-dbncz"]
Sep 10 17:59:47.994913 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.994872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80181a5e-56eb-41cb-ad61-565a9e28e865-must-gather-output\") pod \"must-gather-dbncz\" (UID: \"80181a5e-56eb-41cb-ad61-565a9e28e865\") " pod="openshift-must-gather-j8jqk/must-gather-dbncz"
Sep 10 17:59:47.995108 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:47.994960    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tpn\" (UniqueName: \"kubernetes.io/projected/80181a5e-56eb-41cb-ad61-565a9e28e865-kube-api-access-q2tpn\") pod \"must-gather-dbncz\" (UID: \"80181a5e-56eb-41cb-ad61-565a9e28e865\") " pod="openshift-must-gather-j8jqk/must-gather-dbncz"
Sep 10 17:59:48.095993 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:48.095955    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80181a5e-56eb-41cb-ad61-565a9e28e865-must-gather-output\") pod \"must-gather-dbncz\" (UID: \"80181a5e-56eb-41cb-ad61-565a9e28e865\") " pod="openshift-must-gather-j8jqk/must-gather-dbncz"
Sep 10 17:59:48.096157 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:48.096074    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tpn\" (UniqueName: \"kubernetes.io/projected/80181a5e-56eb-41cb-ad61-565a9e28e865-kube-api-access-q2tpn\") pod \"must-gather-dbncz\" (UID: \"80181a5e-56eb-41cb-ad61-565a9e28e865\") " pod="openshift-must-gather-j8jqk/must-gather-dbncz"
Sep 10 17:59:48.096395 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:48.096377    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80181a5e-56eb-41cb-ad61-565a9e28e865-must-gather-output\") pod \"must-gather-dbncz\" (UID: \"80181a5e-56eb-41cb-ad61-565a9e28e865\") " pod="openshift-must-gather-j8jqk/must-gather-dbncz"
Sep 10 17:59:48.109669 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:48.109637    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tpn\" (UniqueName: \"kubernetes.io/projected/80181a5e-56eb-41cb-ad61-565a9e28e865-kube-api-access-q2tpn\") pod \"must-gather-dbncz\" (UID: \"80181a5e-56eb-41cb-ad61-565a9e28e865\") " pod="openshift-must-gather-j8jqk/must-gather-dbncz"
Sep 10 17:59:48.209146 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:48.209065    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8jqk/must-gather-dbncz"
Sep 10 17:59:48.481290 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:48.481264    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8jqk/must-gather-dbncz"]
Sep 10 17:59:48.483419 ip-10-0-142-242 kubenswrapper[2568]: W0910 17:59:48.483388    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80181a5e_56eb_41cb_ad61_565a9e28e865.slice/crio-d61985b79b65961181732bb03ce1a426d3ab6b25f90dc6fc1175dd2733a72db7 WatchSource:0}: Error finding container d61985b79b65961181732bb03ce1a426d3ab6b25f90dc6fc1175dd2733a72db7: Status 404 returned error can't find the container with id d61985b79b65961181732bb03ce1a426d3ab6b25f90dc6fc1175dd2733a72db7
Sep 10 17:59:48.485223 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:48.485206    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:59:48.577822 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:48.577784    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8jqk/must-gather-dbncz" event={"ID":"80181a5e-56eb-41cb-ad61-565a9e28e865","Type":"ContainerStarted","Data":"d61985b79b65961181732bb03ce1a426d3ab6b25f90dc6fc1175dd2733a72db7"}
Sep 10 17:59:49.585609 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:49.585514    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8jqk/must-gather-dbncz" event={"ID":"80181a5e-56eb-41cb-ad61-565a9e28e865","Type":"ContainerStarted","Data":"554fbbbc5298aadc8fd80138c8738b29a29a9b7dd069b3b49547738210dd6b6c"}
Sep 10 17:59:49.585609 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:49.585564    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8jqk/must-gather-dbncz" event={"ID":"80181a5e-56eb-41cb-ad61-565a9e28e865","Type":"ContainerStarted","Data":"e9cb72c31a7c1431414e576cf91b5b715e7742f9430dd3473e66b2f62a402149"}
Sep 10 17:59:49.605252 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:49.605166    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8jqk/must-gather-dbncz" podStartSLOduration=1.797339553 podStartE2EDuration="2.605145089s" totalImagesPullingTime="807.805536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:59:49.602906391 +0000 UTC m=+2158.312500668" watchObservedRunningTime="2026-09-10 17:59:49.605145089 +0000 UTC m=+2158.314739357"
Sep 10 17:59:50.837620 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:50.837590    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5gjsv_ced14a44-5fd5-4179-af31-eef5392a3e7c/global-pull-secret-syncer/0.log"
Sep 10 17:59:51.044229 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:51.044102    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pntqj_8dfc98be-bcb2-44bd-a4eb-fe5cab6d6571/konnectivity-agent/0.log"
Sep 10 17:59:51.112780 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:51.112703    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-242.ec2.internal_1f3333ec3c604b826ba36ee4849cfacb/haproxy/0.log"
Sep 10 17:59:55.922364 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:55.922335    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-887dw_b0a3af90-b4f8-4610-8c85-c2e6603d1a67/manager/0.log"
Sep 10 17:59:55.952248 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:55.952186    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-6qqc6_9acead52-cfd6-48ae-9d7e-89cc1e472a93/manager/0.log"
Sep 10 17:59:56.043661 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:56.043605    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gzb7q_b8b75e63-9a4e-475a-b9bf-bb0f8c5bc141/manager/0.log"
Sep 10 17:59:56.141970 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:56.141876    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-mtmz2_e357f2b7-993e-433c-bc09-5d2e65445339/manager/0.log"
Sep 10 17:59:57.755180 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:57.755109    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-kf66d_c1a2785a-f374-4ccd-8c7b-249bc472a1c7/kube-state-metrics/0.log"
Sep 10 17:59:57.777638 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:57.777599    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-kf66d_c1a2785a-f374-4ccd-8c7b-249bc472a1c7/kube-rbac-proxy-main/0.log"
Sep 10 17:59:57.821269 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:57.821222    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-kf66d_c1a2785a-f374-4ccd-8c7b-249bc472a1c7/kube-rbac-proxy-self/0.log"
Sep 10 17:59:58.191220 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:58.191193    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmfzv_f1adb054-3538-436e-af61-d94dddb380f4/node-exporter/0.log"
Sep 10 17:59:58.215200 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:58.215109    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmfzv_f1adb054-3538-436e-af61-d94dddb380f4/kube-rbac-proxy/0.log"
Sep 10 17:59:58.242728 ip-10-0-142-242 kubenswrapper[2568]: I0910 17:59:58.242603    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmfzv_f1adb054-3538-436e-af61-d94dddb380f4/init-textfile/0.log"
Sep 10 18:00:00.815309 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:00.815268    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"]
Sep 10 18:00:00.823470 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:00.823438    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:00.827198 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:00.827115    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"]
Sep 10 18:00:00.925265 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:00.924699    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npdjs\" (UniqueName: \"kubernetes.io/projected/7f657bc9-aebb-4149-abf0-e0abcf113502-kube-api-access-npdjs\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:00.925265 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:00.924770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-lib-modules\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:00.925265 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:00.924837    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-sys\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:00.925265 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:00.924898    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-proc\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:00.925265 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:00.924936    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-podres\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026152 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026117    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-sys\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026354 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026171    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-proc\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026354 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026269    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-sys\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026354 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026274    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-proc\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026354 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026296    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-podres\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026587 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026364    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npdjs\" (UniqueName: \"kubernetes.io/projected/7f657bc9-aebb-4149-abf0-e0abcf113502-kube-api-access-npdjs\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026587 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026406    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-lib-modules\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026587 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026414    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-podres\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.026587 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.026552    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f657bc9-aebb-4149-abf0-e0abcf113502-lib-modules\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.035457 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.035429    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npdjs\" (UniqueName: \"kubernetes.io/projected/7f657bc9-aebb-4149-abf0-e0abcf113502-kube-api-access-npdjs\") pod \"perf-node-gather-daemonset-fmnxv\" (UID: \"7f657bc9-aebb-4149-abf0-e0abcf113502\") " pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.140283 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.140173    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.423909 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.423836    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6556d7688-dch9k_3e8edf05-b178-4a63-9548-42707aca3656/console/0.log"
Sep 10 18:00:01.438475 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.438332    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"]
Sep 10 18:00:01.441542 ip-10-0-142-242 kubenswrapper[2568]: W0910 18:00:01.441511    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f657bc9_aebb_4149_abf0_e0abcf113502.slice/crio-2a672c83123051eb664b1baef47fc3e66a7d4616d87fa5b35cc126d3e5194901 WatchSource:0}: Error finding container 2a672c83123051eb664b1baef47fc3e66a7d4616d87fa5b35cc126d3e5194901: Status 404 returned error can't find the container with id 2a672c83123051eb664b1baef47fc3e66a7d4616d87fa5b35cc126d3e5194901
Sep 10 18:00:01.698054 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.697964    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv" event={"ID":"7f657bc9-aebb-4149-abf0-e0abcf113502","Type":"ContainerStarted","Data":"64f41b74544147aedbad5b5a07e29b96376bea82e24f1ee4da3f6045182ffaf1"}
Sep 10 18:00:01.698054 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.698023    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv" event={"ID":"7f657bc9-aebb-4149-abf0-e0abcf113502","Type":"ContainerStarted","Data":"2a672c83123051eb664b1baef47fc3e66a7d4616d87fa5b35cc126d3e5194901"}
Sep 10 18:00:01.698335 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.698161    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:01.718127 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:01.718073    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv" podStartSLOduration=1.718056989 podStartE2EDuration="1.718056989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 18:00:01.715943371 +0000 UTC m=+2170.425537639" watchObservedRunningTime="2026-09-10 18:00:01.718056989 +0000 UTC m=+2170.427651256"
Sep 10 18:00:02.183454 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:02.183415    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-bxzh2_6946ebd9-258c-40f2-ac75-fe7e99aafb1b/volume-data-source-validator/0.log"
Sep 10 18:00:03.385278 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:03.385228    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lqdv_412fe3b2-c273-4b82-b8a9-cf0559396157/dns/0.log"
Sep 10 18:00:03.407683 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:03.407651    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lqdv_412fe3b2-c273-4b82-b8a9-cf0559396157/kube-rbac-proxy/0.log"
Sep 10 18:00:03.479748 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:03.479722    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2pdpk_f39fb70c-4b7c-4983-a5a1-9ed4d9fa1d54/dns-node-resolver/0.log"
Sep 10 18:00:04.101487 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:04.101456    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rtwg_63e68c07-a8f9-4182-a40c-a1ef1470e7cc/node-ca/0.log"
Sep 10 18:00:05.565519 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:05.565492    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf54bbc88-h467l_1654d45e-f4d6-4a2f-9c4c-f72633158a71/kube-auth-proxy/0.log"
Sep 10 18:00:05.653301 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:05.653264    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-s27sr_008bb171-a493-4df9-b15e-e2d8a23ad60e/payload-pre-processing/0.log"
Sep 10 18:00:06.659059 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:06.659027    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhlnf_945e40c2-0163-4303-98ca-1519c942c638/serve-healthcheck-canary/0.log"
Sep 10 18:00:07.318753 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:07.318723    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mkd2r_b05fb386-89c1-46e6-84c0-e494741965e1/kube-rbac-proxy/0.log"
Sep 10 18:00:07.339936 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:07.339904    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mkd2r_b05fb386-89c1-46e6-84c0-e494741965e1/exporter/0.log"
Sep 10 18:00:07.361023 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:07.360993    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mkd2r_b05fb386-89c1-46e6-84c0-e494741965e1/extractor/0.log"
Sep 10 18:00:07.717561 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:07.717478    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8jqk/perf-node-gather-daemonset-fmnxv"
Sep 10 18:00:09.635293 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:09.635262    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lrltm_ac39f1ad-5fd6-4552-93c9-ffd44795cd5a/manager/0.log"
Sep 10 18:00:09.877454 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:09.877424    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4zr7p_2e7225d6-da90-4e22-88ba-d5bb2556cf79/manager/0.log"
Sep 10 18:00:09.938227 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:09.938104    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c56cf8bbb-b2x2j_79f67499-aa9a-4936-b8a4-ef419a5c065c/manager/0.log"
Sep 10 18:00:09.960767 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:09.960739    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9z5gj_b59c7cdb-b7b3-4acb-b907-76dee58922f2/server/0.log"
Sep 10 18:00:10.282577 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:10.282504    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b6bb7766-xfgqm_763ecc42-f2ff-48ff-ac06-20fa3c0b0856/manager/0.log"
Sep 10 18:00:10.301203 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:10.301168    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b6bb7766-xfgqm_763ecc42-f2ff-48ff-ac06-20fa3c0b0856/copy-manifests/0.log"
Sep 10 18:00:11.680038 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:11.680010    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wrw49_34f39f00-816e-4fde-a4d6-e462637000c9/openshift-lws-operator/0.log"
Sep 10 18:00:11.750730 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:11.750706    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf54bbc88-h467l_1654d45e-f4d6-4a2f-9c4c-f72633158a71/kube-auth-proxy/0.log"
Sep 10 18:00:11.819518 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:11.819486    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-s27sr_008bb171-a493-4df9-b15e-e2d8a23ad60e/payload-pre-processing/0.log"
Sep 10 18:00:13.849435 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:13.849404    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf54bbc88-h467l_1654d45e-f4d6-4a2f-9c4c-f72633158a71/kube-auth-proxy/0.log"
Sep 10 18:00:13.918627 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:13.918599    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-s27sr_008bb171-a493-4df9-b15e-e2d8a23ad60e/payload-pre-processing/0.log"
Sep 10 18:00:16.625681 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:16.625651    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-9l9bf_018f80dc-d002-4a7b-8d8b-c8335aecfa20/migrator/0.log"
Sep 10 18:00:16.645889 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:16.645865    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-9l9bf_018f80dc-d002-4a7b-8d8b-c8335aecfa20/graceful-termination/0.log"
Sep 10 18:00:18.168737 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:18.168710    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2r8ch_5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7/kube-multus-additional-cni-plugins/0.log"
Sep 10 18:00:18.201607 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:18.201558    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2r8ch_5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7/egress-router-binary-copy/0.log"
Sep 10 18:00:18.231860 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:18.231831    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2r8ch_5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7/cni-plugins/0.log"
Sep 10 18:00:18.253432 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:18.253405    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2r8ch_5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7/bond-cni-plugin/0.log"
Sep 10 18:00:18.275921 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:18.275897    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2r8ch_5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7/routeoverride-cni/0.log"
Sep 10 18:00:18.298455 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:18.298433    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2r8ch_5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7/whereabouts-cni-bincopy/0.log"
Sep 10 18:00:18.321358 ip-10-0-142-242 kubenswrapper[2568]: I0910 18:00:18.321328    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2r8ch_5b9997a7-c41e-4004-9f12-a6bc9bfe4ec7/whereabouts-cni/0.log"